Linux Networking breaks on Proxmox VM with VirtIO

[Robin-Rhee]

In my experiment, I compiled the source code to form a native environment
So that I created the following veth NICs and network namespaces.

ip netns add ns-ran
ip link add veth-core type veth peer name veth-ran
ip link set veth-ran netns ns-ran
ip addr add 192.168.254.2/24 dev veth-core
ip link set veth-core up
ip netns exec ns-ran ip addr add 192.168.254.1/24 dev veth-ran
ip netns exec ns-ran ip link set veth-ran up
ip netns exec ns-ran ip link set lo up

It's a one in one out setup based on an Ubuntu 24.04 Noble VM
I supposed the veth driver cannot support XDP
So the xdp field is set as "generic"
Here is the complete core config

logging:
  system:
    level: "debug"
    output: "stdout"
  audit:
    output: "stdout"
db:
  path: "core.db"
interfaces:
  n2:
    name: "veth-core"
    port: 38412
  n3:
    name: "veth-core"
  n6:
    name: "enp6s18"
  api:
    name: "enp6s18"
    port: 9999
xdp:
  attach-mode: "generic"
telemetry:
  enabled: false
  otlp-endpoint: "localhost:4317"

However, I have tested the zero entrypoint approach
but the result proved that the NICs cannot support native mode

And here is the database
db.zip

UERANSIM is able to establish the PDU session
But when it starts to ping the internet
The OS paralyzed totally without any reaction
Can't even receive interrupt signal
The only way is to reboot

Here are some log screenshots at the very moment of deadlock

[ RAN ]

[ Core ]

[ UE ]

Here is the pcap captured from veth-core
But it's weird to observe that the signaling procedure seems lacking integrity
though the UERANSIM logs quite intactly

pcap.zip

Any idea on the phenomenon, folks ?

[gruyaume]

Hello @Robin-Rhee, thanks for opening this discussion.

It looks like you are locking yourself out of the VM, but I'm not quite sure why.

@ghislainbourgeois do you see what I'm missing?

[Robin-Rhee]

Dockerized setup no problem.
Next step I'll move on to form and test a USRP setup (substituting UERANSIM). 😉😉

[Robin-Rhee]

Got a new observation which sounded the alarm
If I launch a native setup, ping, and tear it down
then start a dockerized setup
the dockerized setup failed to ping 😂😂

This phenomenon harassed me when I was building the docker image of ella core after tearing down the test environment of the native setup
I observed that the RUN apt-get update command in the Dockerfile always failed to reach the internet with continuous "Ign: xxx (apt sources)"
which indicated that docker got a problem in connecting the networks.
Then I reboot the system to successfully recover the networks.
Such a situation can be avoided by always using only one kind of setup though
You guys should be alert to the concern that something uncertain may undermine the network stack covertly
which may furtherly compromise the reliability of your system in the future

[Robin-Rhee]

Failed to pass USRP/COTS_UE (Pixel 6) test.
Phenomenon weird
A lot of small flaws can be left out for now
including several derived from Free5gc
But that's OK
Not a big deal
The most important thing is UE can ping
DNS no problem either from the perspective of terminal
But Web pages cannot be accessed
This situation usually can be solved by shutting the Pixel hidden switches by entering *#*#0702#*#*
in a way that overlooks the IMS registration process
BTW, here are the pcaps
pcap.zip
Basically, PDU sessions have been successfully established.
Due to lack of IMS, the UE still tends to release the N1 connection.
which usually can be solved by turning on IMS VoPS switch in Registration Accept signaling
and toggling off the aforementioned Pixel hidden switches
Need more debugging
especially the logic of logs and pcaps looked pretty messed up
Have been debugging for day and night
So freaking exhausted
Have to go home to sack out for a while
C U 🐱‍🏍🐱‍🏍

[Robin-Rhee]

I've rebased and re-compiled the source code
Re-write the USIM card with a new IMSI
Update the code db
Now it seems to behave correctly
But the Internet connection is not quite stable
some websites cannot be accessed
and it still cut off N1 reference point after a certain time (180s+)
The magic switches of Pixel didn't work too much
Maybe you guys one day consider opening N5 reference point of PCF
So that the PCSCF is able to fit in
But it significantly increases the complexity of the system
That may probably not be what you want
However, ella core really is a groundbreaking art that
streamlines the deployment and operation of 5G networks
Good job, guys
I'll stay tuned with your improvements 😄😄

[Robin-Rhee]

I use nohup to keep core and gnb alive in the background
It proved that the *#*#0702#*#* magic switches actually take effects 😎😎👏👏

[Robin-Rhee]

Summary:

1. Use intel e1000 network driver instead of virtio in a VM running on Proxmox
2. Changing from native setup to a dockerized setup need a reboot to enable the network stack
3. Use the *#*#0702#*#* magic switches to enable a Pixel phone to maintain the Data session without an IMS session established

[Robin-Rhee]

@gruyaume @ghislainbourgeois
Hi guys

I came back to make a Proxmox test VM with Ella co-hosting with OCUDU
by referring to https://docs.ellanetworks.com/how_to/co_host_with_ocudu
I tried to use veth peers residing in different namespaces as endpoints of N2/N3
which still suffered from the aforementioned problem in this thread.

PDU session could be successfully established
I logged in the shell of UE via ADB

Pinged the Internet successfully via ping -I rmnet1 8.8.8.8

No problem with DNS resolution

But it failed to access Internet through a browser
So I supposed the networking is good, but the user plane based on eBPF/XDP is broken
by referring to https://docs.ellanetworks.com/explanation/user_plane_packet_processing_with_ebpf

Then I gave up the veth approach, and embraced the physical NIC method
in which I created 2 e1000 NICs (192.168.254.1/30 & 192.168.254.2/30) for the same VM

Still allocate them to different namespaces respectively
with 2 USRP PCIe-passthrough NICs too (192.168.253.1/24 & 192.168.252.1/24)

#!/bin/bash

set -x

ip netns add ns-ran
ip netns exec ns-ran ip link set lo up
ip link set eth2 netns ns-ran
ip netns exec ns-ran ip addr add 192.168.254.2/30 dev eth2
ip netns exec ns-ran ip link set eth2 up
ip link set eth3 netns ns-ran
ip netns exec ns-ran ip addr add 192.168.253.1/24 dev eth3
ip netns exec ns-ran ip link set eth3 up
ip link set eth4 netns ns-ran
ip netns exec ns-ran ip addr add 192.168.252.1/24 dev eth4
ip netns exec ns-ran ip link set eth4 up

Finally the Internet can be accessed in the browser

I have read through those guides on eBPF and XDP in the Ella portal
but failed to find too much instructive materials on the realization of "generic" mode of XDP attachment as well as the difference between veth peers deployment and physical NICs deployment

Could you be so kinds as to elaborate or supplement the knowledge base about the following topics ?

• "generic" mode of XDP attachment, and the difference between it and "xdp" mode
• Step by step to make sure the NIC supports XDP, though I saw the list of drivers and kernel versions you referred to
• Difference between co-host deployment (5GC and gNB on the same host) and separate deployment (5GC and gNB on 2 hosts)

And maybe in an appropriate time being, you would fix the problem 😄😄

[Robin-Rhee]

@gruyaume
Hi, bro

I have disabled kernel offloads and retest
But the web page still times out
Ping test also fails with packet size larger than 200 bytes

# ethtool -k eth0 | grep -E "gro|gso|tso"
tx-gso-robust: off [fixed]
tx-gso-partial: off [fixed]
tx-gso-list: off [fixed]
rx-gro-hw: off [fixed]
rx-gro-list: off
rx-udp-gro-forwarding: off

BTW, the traffic page has any help ?

I have to doze off for a moment, bro
Leave me messages if there is anything I can help
I'll be back as soon as possible 😉😉

[gruyaume]

Hello @Robin-Rhee ,

I tried to reproduce this issue without success. I have a setup with a raspberry Pi, exactly as documented in the Co-host with OCUDU guide. I have a Galaxy A56 connected and browsing works fine and we can ping 1400 bytes packets from the UE. The fact that yours caps at 200 bytes is strange.

Environment details:

• Raspberry Pi 5 (baremetal)
• Kernel: 6.8.0-2038-raspi-realtime
• Ella Core: v1.14

A couple of differences from your setup that migh matter:

• N2 and N3 are separate here (N2 on 10.3.0.2, N3 on the veth). In your case they are both on veth-core.
• You're on a KVM VM, we're on baremetal. The virtual NIC driver can matter for XDP. Could you share ethtool -i eth0?
• You're using an old binary it seems.

I'm not quite sure what's the root cause of your issue, I'll wait for @ghislainbourgeois to come back. He may have more ideas.

[Robin-Rhee]

Hi @gruyaume
You're absolutely sharp !!
With respect to your concerns

• I don't quite actually wanna change the current situation in which N2 and N3 reside on the same veth, so let's handle the other 2 concerns firstly

• The KVM VM is based on the platform Proxmox, this situation won't change too. Actually, I have another USRP setup based on a VM leveraging Open5gs/Free5gc. They both tend to use the Linux network stack and iptables to handle forwarding. Those setups work maturely but in a way that can handle less throughput. You guys choose an advanced method -- eBPF/XDP, for which I feel great admiration. I love the all-in-one design of Ella core so much so that hope you guys make it for virtual environment too.

Here is the info of eth0:

# ethtool -i eth0
driver: e1000
version: 6.8.0-136-generic
firmware-version:
expansion-rom-version:
bus-info: 0000:06:12.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Actually in the first place of current thread, the eth0 was based on virtio driver which led to crashes of Linux network stack. Then you know, @ghislainbourgeois figured out the problem with virtio, then I switch the driver to e1000, which had been tested successfully after that. So I may prefer to suppose that the current version of Ella I use may have some gaps to fill.

• Based on the previous reflection, I plan to git pull the latest Ella source code and build the latest version 1.14 to give a try. Wait for my message, bro. 😉😉

[Robin-Rhee]

@gruyaume
Hi dude,

I have updated the source code to 1.14.0 and rebuilt the project by go1.26.5

But it seems the problem still exists

BTW, I reused the db file. I supposed it should be OK, right ?

FYI, this is the kernel and OS info of the building VM (another VM)

# uname -r
6.17.0-20-generic

# hostnamectl
 Static hostname: noble
       Icon name: computer-vm
         Chassis: vm
      Machine ID: 4cf12b39082f447c933c765dfcb5bf89
         Boot ID: 24c06504fc204d008e7df8cdd89e8274
  Virtualization: kvm
Operating System: Ubuntu 24.04.4 LTS
          Kernel: Linux 6.17.0-20-generic
    Architecture: x86-64
 Hardware Vendor: QEMU
  Hardware Model: Standard PC _Q35 + ICH9, 2009_
Firmware Version: rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org
   Firmware Date: Tue 2014-04-01
    Firmware Age: 12y 3month 3w 1d

[gruyaume]

I deployed Ella Core in a VM environment here and I was unable to reproduce the issue.

Environment:

• Multipass KVM/QEMU VM (Ubuntu 24.04.4, kernel 6.8.0-134-generic, x86-64)
• Ella Core snap v1.14.0
• Generic XDP mode
• N2+N3 combined on a veth pair (core↔ran-veth in netns ran)
• N6 on the VM's virtio-net NIC (ens3)
• UERANSIM as the gNB+UE

I had to disable TX checksum offload on the veth pair (ethtool -K core tx off and ethtool -K ran-veth tx off) to get the UE to get internet connectivity (ICMP at all sizes and TCP)

[gruyaume]

Hi @Robin-Rhee, I reproduced your exact issue on a QEMU/KVM VM with an e1000 NIC (small packets fine, anything past ~200 bytes fails). I think we found the root cause (thanks to AI).

The fix

Disable TX-checksum offload on both ends of your N3 veth pair:

sudo ethtool -K veth-core tx off
sudo ip netns exec ns-ran ethtool -K veth-ran tx off

What's going on

When a program sends traffic over a veth, Linux normally skips computing the packet's checksum and leaves a note saying "the network card will finish this checksum at byte X later." So your gNB's GTP packets reach Ella Core with an incomplete checksum plus that note.

Ella Core's user plane strips the outer GTP header (in XDP) but Linux's generic-XDP path doesn't update that "finish at byte X" note when it shifts the packet forward. The note now points at the wrong place. When the packet finally leaves the NIC, the card writes a checksum at that wrong spot and mangles the packet, so it gets dropped downstream.

Why small packets survive but large ones don't on e1000: the stale "byte X" lands past the end of short packets (card leaves them alone) but inside longer ones (card corrupts them), that's your ~200-byte cliff. On a virtio NIC the same problem breaks every size.

Whose issue is this?

It's a gap in Linux's generic (SKB-mode) XDP: the kernel shifts the packet but never updates the checksum note — not Ella's own logic, and it's triggered by a checksum-offload setting on your gNB's veth, which Ella doesn't control. It seems to be a known sharp edge in the XDP world: Cilium's and the XDP maintainers' own test scripts disable checksum offload on veths because XDP-forwarded packets otherwise end up with invalid checksums. Two things make it disappear:

• disabling the veth's TX-checksum offload (the command above), or
• switching to native XDP mode, which forwards raw frames without the checksum-offload metadata that causes this — though the e1000 driver doesn't support native XDP, so that isn't an option on your NIC.

Since your NIC can't do native XDP, the ethtool command is your path. Give it a try and let us know if your browser works afterwards, it did in our reproduction. Happy to share the deeper kernel-level details if you're curious.

[gruyaume]

Hello @Robin-Rhee ,

I'm glad it works!

Here's Claude's answer to your questions:

On native XDP

The "raw bytes, no SKB" picture holds for a physical NIC, where XDP runs on the raw frame before the kernel builds a socket buffer. A veth — which is what Ella Core decaps on here — works differently: the packet is an sk_buff from the moment the sender creates it, and even native XDP runs on that same skb, so the CHECKSUM_PARTIAL note travels with it. That note comes from the sender's TX offload, and no XDP mode clears it on a veth — which is why the real fix is ethtool -K … tx off, stopping the note being created in the first place.

On namespaces

When both endpoints sit in the same namespace, each IP is local there, so the kernel delivers between them over loopback — the packets never traverse the veth, so Ella Core's XDP never sees them and never decaps. A bare ping between the two IPs would actually work (local delivery); it's the end-to-end UE datapath that fails, because it needs those tunnel packets to physically cross the veth. Separate namespaces make each address non-local, forcing the traffic onto the veth where XDP can process it.

[Robin-Rhee]

@gruyaume
Thanks a lot for detailed clarification, my friend !!

I've got no problem with the namespace aspect

For the XDP aspect
I wanna confirm the nuances between generic XDP & native XDP
Could you be kind enough to make sure I understand the process in the right way 😁🤞🏻

generic XDP (XDPGenericMode)

• Ella's N3 RX veth receives the SKB with stale metadata intact
• N3 RX veth passes SKB pointer to Ella Core.
• Ella's veth XDP hook executes eBPF program, strips GTP header in remove_gtp_header function
• Generic XDP modifies the packet bytes but does NOT clear or update the stale CHECKSUM_PARTIAL metadata
• Generic XDP issues bpf_redirect() which returns XDP_REDIRECT to the N6 physical NIC
• The N6 physical NIC driver sees skb->ip_summed == CHECKSUM_PARTIAL and trusts it, finally corrupts the packet by computing the checksum and write it at csum_start + csum_offset, which now points inside the inner IP/UDP payload

native XDP (XDPDriverMode) on a compatible NIC driver

• Ella's N3 RX veth receives the SKB with stale metadata intact
• N3 RX veth passes SKB pointer to Ella Core.
• Ella's veth XDP hook executes eBPF program, strips GTP header in remove_gtp_header function
• Native XDP modifies the packet bytes but does NOT clear or update the stale CHECKSUM_PARTIAL metadata
• Native XDP issues bpf_redirect() which returns XDP_REDIRECT to the N6 physical NIC
• Native XDP check if the N6 physical NIC has attached a customized driver, then consults the customized driver zero_entrypoint.o if the packet can be granted admission XDP_PASS. If either condition doesn't meet, drop it.
• If the N6 physical NIC driver supports native XDP (XDPDriverMode), the kernel bypass creating a new SKB for TX.
• The N6 physical NIC TX ring buffer receives the raw frame pointer without an SKB attached
• The N6 physical NIC transmits the raw bytes onto the wire. The inner frame already has a full checksum computed by UE

[Robin-Rhee]

Hi @gruyaume
I saw there is a description in the documentation that the customized driver zero_entrypoint.o will be applied to the veth pair

But I checked out the source code in internal\upf\upf.go as follows

	n3Link, err := link.AttachXDP(link.XDPOptions{
		Program:   bpfObjects.UpfEntryFunc,
		Interface: n3Iface.Index,
		Flags:     StringToXDPAttachMode(xdpAttachMode),
	})
	if err != nil {
		return nil, fmt.Errorf("failed to attach eBPF program on n3 interface %q: %s", n3AttachmentInterface, err)
	}

	var n6Link *link.Link

	if n6Iface.Index != n3Iface.Index {
		n6, err := link.AttachXDP(link.XDPOptions{
			Program:   bpfObjects.UpfEntryFunc,
			Interface: n6Iface.Index,
			Flags:     StringToXDPAttachMode(xdpAttachMode),
		})
		if err != nil {
			return nil, fmt.Errorf("failed to attach eBPF program on n6 interface %q: %s", n6AttachmentInterface, err)
		}

		n6Link = &n6
	}

I noted the XDP programs are designated to be attached to Ella N3 interface and N6 interface
in a way that agrees to the following illustration

instead of being attached to both endpoints of N3 veth pair as the documentation describes
I dunno if the captions need to be adjusted or if I understand it mistakenly

BTW, I really love the way you elaborate the mindsets under the hood of developing Ella Core in the documentation
And the impressive watchword "implicit knowledge is a liability to minimize." 😉😉

[gruyaume]

Hello @Robin-Rhee ,

Thanks for the interest.

I had Claude reply to your answer by referring to Ella Core's codebase and Linux itself. Here's what it had to say.

• Generic XDP. The gNB's packet crosses the veth as an skb carrying the CHECKSUM_PARTIAL note. Ella Core's user plane program runs on that skb, strips the GTP-U header, and returns XDP_REDIRECT. The kernel adjusts the skb's data pointer but leaves csum_start/csum_offset untouched, and the packet stays an skb all the way out to the N6 driver — nothing on that path invalidates or fixes the note after the decap. The NIC then trusts the stale note and writes a checksum at a bogus offset, corrupting the frame. That's the corruption you hit, and why ethtool -K … tx off fixes it.

• Native XDP on a veth. The packet still arrives as an skb with the same stale note, and the program does the same decap. The difference is at redirect: the kernel consumes the skb and converts the packet to a raw xdp_frame, a structure with no checksum fields at all — the metadata is discarded right there. The frame goes straight to the N6 driver's ndo_xdp_xmit and is transmitted byte-for-byte as the program left it. Since the inner packet's checksums were already computed by the UE, it goes out clean.

On zero_entrypoint.o: it isn't involved on N6, and it doesn't filter anything — it unconditionally passes. It exists because the veth driver only delivers natively-redirected frames (downlink, N6 → N3) when the receiving peer's NAPI is active, and loading any XDP program on the peer guarantees that (on recent kernels, ethtool -K veth-ran gro on after link-up works too). Physical NICs have an analogous requirement — their XDP TX rings only exist once a program is attached — but on N6 that's already satisfied by Ella Core's own upf_entry program; the target's program is never run on transmitted frames.

That's also the answer to your docs question: both are accurate, they describe different things. upf.go attaches upf_entry to N3 and N6 — that's what the diagram shows. The paragraph you quoted is about the extra manual step for native mode on a veth: the operator attaches zero_entrypoint.o to the RAN-side peer themselves (see Use native XDP with veth interfaces) — Ella Core can't do it, since the peer lives in another namespace. It doesn't apply to generic mode, so it never came into play in your setup.

We are considering adding TCX support, it's a more solid fallback on systems that don't support native XDP. Generic XDP would then become a dev-only option.

[Robin-Rhee]

Fantastic job !! @gruyaume
I try to correct and reframe my words after aligning thoughts with your celestial explanations 😄😄

Native XDP on veth

• The eBPF program runs on the SKB, strips the GTP header, and leaves the packet bytes modified
• Then the program calls bpf_redirect() and returns XDP_REDIRECT
• The kernel takes the SKB and converts it to an xdp_frame — a bare-bones structure that contains only: A pointer to the packet data (raw bytes) and the packet length. No checksum metadata (CHECKSUM_PARTIAL, csum_start, and csum_offset)
• The kernel calls the NIC driver's ndo_xdp_xmit() function, which takes the xdp_frame directly -> places the raw packet data into the NIC's transmit ring buffer -> does not look at any SKB metadata because there is no SKB
• The NIC hardware transmits the raw bytes outbound exactly as they are in memory — byte-for-byte as the eBPF program left them

External & internal eBPF/XDP program

• It is not a filter or a processing program. It’s a dummy XDP program that simply passes all packets by returning XDP_PASS
• The veth driver’s native XDP redirection (downlink from N6 -> N3) only works if the receiving peer’s NAPI is active/scheduled
• Native XDP redirect from N6 to N3 will silently drop packets if NAPI of N3 interface is not active/scheduled
• Attaching any XDP program to that peer guarantees NAPI is running — hence we need the dummy program
• We must manually attach it to the endpoints of veth pair. Ella Core cannot do this because it doesn’t have visibility into different namespaces.
• ethtool -K veth-ran gro on after link-up can also activate NAPI without an XDP program
• A similar requirement exists for the physical NIC — XDP TX rings are only created when an external XDP program is attached
• On the N6 interface, this is already satisfied because Ella Core’s own upf_entry program is attached to the N6 NIC.
• Only the downlink needs the external XDP program because the uplink has been fulfilled by Ella Core’s own upf_entry program which applies to both N3 and N6 of Ella Core

NAPI

• NAPI (New API) is the Linux networking mechanism that interrupts + polls for incoming packets in a way that is more efficient for high-speed traffic
• In the working state, NAPI disables interrupts on the RX queue of an interface -> polls the queue in a loop -> re-enables interrupts when polling is done
• To activate the NAPI, the interface's NAPI structure is initialized by netif_napi_add() -> the NAPI instance is set to a NAPI_STATE_SCHED state by napi_schedule() -> the NAPI poll function poll() is running or ready to run

veth activates NAPI

• The logic is defined in the veth driver (drivers/net/veth.c):
• The XDP redirect path veth_xdp_rcv() requires the receiving peer's NAPI to be scheduled.
• If the peer has no XDP program attached, its NAPI cannot be scheduled — so redirect packets are dropped.
• Attaching zero_entrypoint.o forces veth_napi_enable() to be called, which schedules NAPI and it's ready to poll
• The veth driver checks napi_schedule_prep() or similar internally to make sure NAPI is active/scheduled