Open5GS + multiple load balanced eUPF with BGP and NAT

[infinitydon]

Hi,

I just tried to test a way for solving the multi eUPFs using the same IP pool.

Architecture is below

A single FRR is used with a unique VRF for each of the eUPFs.

FRR POD startup commands (the VRF CNI plugin can also be used to reduce the command snippets):

ip link add dev eupf1 type vrf table 2;
ip link add dev eupf2 type vrf table 3;
ip link set dev eupf1 up;
ip link set dev eupf2 up;
ip link set dev n6-upf1 master eupf1;
ip link set dev n6-upf2 master eupf2;
ip addr add 192.168.101.1/24 dev n6-upf1;
ip addr add 192.168.101.2/24 dev n6-upf2;
ip route add table 2 0.0.0.0/0 via 192.168.101.254;
ip route add table 3 0.0.0.0/0 via 192.168.101.254;
echo 1 > /proc/sys/net/ipv4/ip_forward;
ip vrf exec eupf1 iptables -t nat -A POSTROUTING -s 10.11.0.0/16 -o n6-upf1 -j MASQUERADE; 
ip vrf exec eupf2 iptables -t nat -A POSTROUTING -s 10.11.0.0/16 -o n6-upf2 -j MASQUERADE; 
touch /etc/frr/vtysh.conf;
sed -i 's/bgpd=no/bgpd=yes/g' /etc/frr/daemons;
/usr/bin/tini --;
/usr/lib/frr/docker-start

FRR BGP config:

router bgp 65001 vrf eupf1
 neighbor 192.168.101.3 remote-as 65001
!
router bgp 65001 vrf eupf2
 neighbor 192.168.101.4 remote-as 65001
!      
line vty
!
end

eUPF1 helm config:

command: ["/bin/sh", "-c"]

args: [
   mkdir -p /etc/iproute2/;
   echo '1000 n6if' >> /etc/iproute2/rt_tables;
   ip rule add from 10.11.0.0/16 table n6if;
   ip route add default via 192.168.101.1 dev n6 table n6if;
   sh /app/bin/entrypoint.sh --config /app/conf/config.yml
 ]
configMaps:
  config:
    data:
      config.yml: |
        interface_name: [eth0, n6]
        api_address: :8080
        pfcp_address: :8805
        metrics_address: :9090

volumes:
  - name: sys
    hostPath:
      path: /sys
  - name: config
    configMap:
      name: eupf1-config
      defaultMode: 420
      items:
        - key: config.yml
          mode: 493
          path: config.yml

volumeMounts:
  - name: sys
    mountPath: /sys
    readOnly:  true
  - name: config
    mountPath: /app/conf

bird:
  enabled: true
  image: docker.io/edgecom/bird:0.0.4b
  env:
    BIRD_LOCAL_AS: 65001
    BIRD_LOCAL_PREFIX: 10.11.0.0/16
    BIRD_NEIGHBOR_IP: 192.168.101.1
    BIRD_NEIGHBOR_AS: 65001

podAnnotations:
  k8s.v1.cni.cncf.io/networks: >-
    [
    { "name": "n6-network", "interface": "n6", "ips": ["192.168.101.3/24"] }    
    ]

eUPF2 helm config:

command: ["/bin/sh", "-c"]

args: [
   mkdir -p /etc/iproute2/;
   echo '1000 n6if' >> /etc/iproute2/rt_tables;
   ip rule add from 10.11.0.0/16 table n6if;
   ip route add default via 192.168.101.2 dev n6 table n6if;
   sh /app/bin/entrypoint.sh --config /app/conf/config.yml
 ]

env:
  UPF_PFCP_NODE_ID: $(MY_POD_IP)
  UPF_N3_ADDRESS: $(MY_POD_IP)

configMaps:
  config:
    data:
      config.yml: |
        interface_name: [eth0, n6]
        api_address: :8080
        pfcp_address: :8805
        metrics_address: :9090

volumes:
  - name: sys
    hostPath:
      path: /sys
  - name: config
    configMap:
      name: eupf2-config
      defaultMode: 420
      items:
        - key: config.yml
          mode: 493
          path: config.yml

volumeMounts:
  - name: sys
    mountPath: /sys
    readOnly:  true
  - name: config
    mountPath: /app/conf

bird:
  enabled: true
  image: docker.io/edgecom/bird:0.0.4b
  env:
    BIRD_LOCAL_AS: 65001
    BIRD_LOCAL_PREFIX: 10.11.0.0/16
    BIRD_NEIGHBOR_IP: 192.168.101.2
    BIRD_NEIGHBOR_AS: 65001

podAnnotations:
  k8s.v1.cni.cncf.io/networks: >-
    [
    { "name": "n6-network", "interface": "n6", "ips": ["192.168.101.4/24"] }    
    ]

FRR route status:

eupf-frr-c68965dc6-cbzq9# show ip route vrf eupf1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF eupf1:
K>* 0.0.0.0/0 [0/0] via 192.168.101.254, n6-upf1, 03:57:28
B>* 10.11.0.0/16 [200/0] via 192.168.101.3, n6-upf1, weight 1, 00:56:20
C>* 192.168.101.0/24 is directly connected, n6-upf1, 03:57:28
eupf-frr-c68965dc6-cbzq9#
eupf-frr-c68965dc6-cbzq9#
eupf-frr-c68965dc6-cbzq9#
eupf-frr-c68965dc6-cbzq9#
eupf-frr-c68965dc6-cbzq9# show ip route vrf eupf2
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF eupf2:
K>* 0.0.0.0/0 [0/0] via 192.168.101.254, n6-upf2, 03:57:32
B>* 10.11.0.0/16 [200/0] via 192.168.101.4, n6-upf2, weight 1, 00:54:08
C>* 192.168.101.0/24 is directly connected, n6-upf2, 03:57:32
eupf-frr-c68965dc6-cbzq9#

Let me know what you think

[pirog-spb]

Hi Christofer @infinitydon

Your proposal looks promising.
Could you please clarify how downlink traffic will be routed back? Do VRFs both use the same NAT instance?

--BR Alex

[infinitydon]

@pirog-spb - Yes, all the VRF router processes and NATs is carried out in the same instance, below shows the NAT config:

ip vrf exec eupf1 iptables -t nat -A POSTROUTING -s 10.11.0.0/16 -o n6-upf1 -j MASQUERADE; 
ip vrf exec eupf2 iptables -t nat -A POSTROUTING -s 10.11.0.0/16 -o n6-upf2 -j MASQUERADE; 

Each VRF instance is mapped to a distinct eUPF N6 address, the eUPF N6 IP will be the next-hop for the return downlink traffic. I also captured this in the route-table of the VRFs:

eupf-frr-c68965dc6-cbzq9# show ip route vrf eupf1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF eupf1:
K>* 0.0.0.0/0 [0/0] via 192.168.101.254, n6-upf1, 03:57:28
B>* 10.11.0.0/16 [200/0] via 192.168.101.3, n6-upf1, weight 1, 00:56:20
C>* 192.168.101.0/24 is directly connected, n6-upf1, 03:57:28
eupf-frr-c68965dc6-cbzq9#
eupf-frr-c68965dc6-cbzq9#
eupf-frr-c68965dc6-cbzq9#
eupf-frr-c68965dc6-cbzq9#
eupf-frr-c68965dc6-cbzq9# show ip route vrf eupf2
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF eupf2:
K>* 0.0.0.0/0 [0/0] via 192.168.101.254, n6-upf2, 03:57:32
B>* 10.11.0.0/16 [200/0] via 192.168.101.4, n6-upf2, weight 1, 00:54:08
C>* 192.168.101.0/24 is directly connected, n6-upf2, 03:57:32
eupf-frr-c68965dc6-cbzq9#