Check all deployment configurations for relevance

[PapaySail]

Need to deploy from scratch as per instructions. See if there are any bugs and what kind of bugs.
Need to strictly follow these instructions https://github.com/edgecomllc/eupf/blob/main/docs%2Fdocs-ru_ru%2Fdeployments%2FREADME.md

Docker-compose deployments:

• Open5GS
• Free5GC
• Free5GC ULCL
• OAI in 5G SA mode with L2 nFAPI

K8s deployments

• Open5GS & Calico BGP
• Open5GS & Calico BGP with slicing
• Open5GS & loadbalancing
• Open5GS & srsRAN & Calico BGP
• Free5GC & Calico BGP
• Free5GC & ULCL
• OAI

[PapaySail]

open5gs-compose

configure firewall `bash fw.sh` flushes rules - it can be harmful
+ Hardcoded `eth0` interface
- and it's not needed for successfull iperf test!

# flush rules
iptables -F
iptables -X
iptables -t raw -F
iptables -t raw -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

Hardcoded eth0 interface

iptables -t nat -A POSTROUTING -s 172.20.0.0/16 -d 0.0.0.0/0 -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.46.0.0/16 -o eth0 -j MASQUERADE

for i in $(seq 5201 5208); do (iperf3 -s -p $i &) ; done

Iperf needed to be installed

sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ for i in $(seq 5201 5208); do (iperf3 -s -p $i &) ; done

sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3

empty env GNB_IP, UE2 can't start

Because of project name is open5gs-compose, env in docker.yaml should be
GNB_HOSTNAME: open5gs-compose-gnb-1

sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ sudo docker-compose -f docker-compose.ue2.yaml logs ue2-03
open5gs-compose-ue2-03-1  | GNB_IP:
open5gs-compose-ue2-03-1  | Launching ue: nr-ue -c ue.yaml
open5gs-compose-ue2-03-1  | UERANSIM v3.2.6
open5gs-compose-ue2-03-1  | terminate called after throwing an instance of 'LibError'
open5gs-compose-ue2-03-1  |   what():  Bad Inet address: null Resource temporarily unavailable
open5gs-compose-ue2-03-1  | /entrypoint.sh: line 41:    14 Aborted                 (core dumped) nr-ue -c ue.yaml $@

run iperf tests unsuccessfull

because of:

• hardcoded IPERF_HOST: 10.128.0.14
• TCP checksum error, see TCP checksum error when connecting from UE to neighbour pod #170

TODO to run iperf test:
Needed to set to IPERF_HOST: 172.20.0.1
sudo ethtool -K br-b8765a6637ca tx off -- for interface --gateway=172.20.0.1

When fixed, iperf tested successfully

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.04  sec   240 MBytes  67.0 Mbits/sec                  receiver
[  5]  29.00-30.00  sec  11.5 MBytes  96.5 Mbits/sec
[  5]  30.00-30.04  sec   324 KBytes  68.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.04  sec   237 MBytes  66.2 Mbits/sec                  receiver
[  5]  30.00-30.04  sec   450 KBytes  84.9 Mbits/sec
[  5]  30.00-30.04  sec   288 KBytes  59.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.04  sec   246 MBytes  68.6 Mbits/sec                  receiver
[  5]   0.00-30.04  sec   252 MBytes  70.3 Mbits/sec                  receiver
[  5]  30.00-30.05  sec   462 KBytes  70.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.05  sec   242 MBytes  67.5 Mbits/sec                  receiver
[  5]  30.00-30.05  sec   710 KBytes   118 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.05  sec   256 MBytes  71.4 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5203
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5206
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5205
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5204
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5202
-----------------------------------------------------------

[PapaySail]

Free5GC & ULCL

make commands

1. make eupf to install eUPF deploy as upfb -- unsuccessfull:

    Events:
     Type     Reason           Age   From               Message
     ----     ------           ----  ----               -------
     Normal   Scheduled        61s   default-scheduler  Successfully assigned free5gculcl/edgecomllc-eupf-6488f49c4b-sn7p7 to edgecom-test-node01
     Warning  SysctlForbidden  61s   kubelet            forbidden sysctl: "net.ipv6.conf.all.forwarding" not allowlisted

✔ Details

sergo@edgecom-test-node01:~$ sudo kubelet --allowed-unsafe-sysctls 'net.ipv4.ip_forward,net.ipv6.conf.all.forwarding'
Flag --allowed-unsafe-sysctls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
I0507 20:27:57.482421   37319 server.go:399] "Kubelet version" kubeletVersion="v1.24.3"
I0507 20:27:57.482516   37319 server.go:401] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""

Solution from manual Reconfiguring the kubelet on a running node

sergo@edgecom-test-node01:~$ sudo cp /etc/kubernetes/kubelet-config.yaml new-kubelet-config.yaml
$ sudo nano new-kubelet-config.yaml
sergo@edgecom-test-node01:~$ sudo kubectl -n kube-system create configmap my-node-config --from-file=new-kubelet-config.yaml --append-hash -o yaml
apiVersion: v1
data:
  new-kubelet-config.yaml: |
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    nodeStatusUpdateFrequency: "10s"
    failSwapOn: True
    authentication:
      anonymous:
        enabled: false
      webhook:
        enabled: True
      x509:
        clientCAFile: /etc/kubernetes/ssl/ca.crt
    authorization:
      mode: AlwaysAllow
    staticPodPath: /etc/kubernetes/manifests
    cgroupDriver: systemd
    containerLogMaxFiles: 5
    containerLogMaxSize: 10Mi
    maxPods: 110
    podPidsLimit: -1
    address: 10.20.253.172
    readOnlyPort: 0
    healthzPort: 10248
    healthzBindAddress: 127.0.0.1
    kubeletCgroups: /systemd/system.slice
    clusterDomain: cluster.local
    protectKernelDefaults: true
    rotateCertificates: true
    clusterDNS:
    - 169.254.25.10
    kubeReserved:
      cpu: 200m
      memory: 512Mi
    resolvConf: "/run/systemd/resolve/resolv.conf"
    allowedUnsafeSysctls:
    - net.ipv4.ip_forward
    - net.ipv6.conf.all.forwarding

    eventRecordQPS: 5
    shutdownGracePeriod: 60s
    shutdownGracePeriodCriticalPods: 20s
kind: ConfigMap
metadata:
  creationTimestamp: "2024-05-08T16:06:30Z"
  name: my-node-config-662mcffb5k
  namespace: kube-system
  resourceVersion: "72162413"
  uid: 0ace9431-2b35-494c-a7db-0139df994ef8
sergo@edgecom-test-node01:~$
sergo@edgecom-test-node01:~$ sudo kubectl edit node edgecom-test-node01
node/edgecom-test-node01 edited

add the following YAML under spec:

  configSource:
    configMap:
        name: my-node-config-662mcffb5k
        namespace: kube-system
        kubeletConfigKey: kubelet

sergo@edgecom-test-node01:~$ export NODE_NAME=edgecom-test-node01
sergo@edgecom-test-node01:~$ sudo kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}"
node/edgecom-test-node01 patched

❌ all above does not work
✔ Ok. Working method:

sudo nano /etc/kubernetes/kubelet-config.yaml 
sudo systemctl restart kubelet

---

2. make upf to install Free5gc UPFs
   CrashLoopBackOff 2024-05-08T17:00:32.904269017Z [ERRO][UPF][Main] UPF Cli Run Error: open Gtp5g: version mismatch: gtp5g version(0.7.3) should be 0.8.1 <= verion < 0.9.0 , please update it

   ✔ Updated gtp5g kernel module

   sudo su
   apt-get update; apt-get install git build-essential -y; \
   cd /tmp; \
   git clone --depth 1 https://github.com/free5gc/gtp5g.git; \
   cd gtp5g/; \
   make && make install
   
   sergo@edgecom-test-node01:/tmp/gtp5g$ sudo modprobe -r gtp5g
   sergo@edgecom-test-node01:/tmp/gtp5g$ sudo lsmod | grep ^gtp
   sergo@edgecom-test-node01:/tmp/gtp5g$ sudo modprobe  gtp5g
   sergo@edgecom-test-node01:/tmp/gtp5g$ sudo lsmod | grep ^gtp
   gtp5g                 126976  0

---

5. make ueransim to install gNodeB and UE simulators.
   UE connects and disconnects

Details

[2024-05-08 17:11:34.717] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]                                                                                                         [2024-05-08 17:11:36.008] [nas] [debug] PDU Session Establishment Accept received                                                                                                                               [2024-05-08 17:11:36.008] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-08 17:11:36.041] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.1.0.1] is up.                                                                             [2024-05-08 17:11:39.012] [rrc] [debug] Signal lost for cell[2], total [0] cells in coverage                                                                                                                    [2024-05-08 17:11:39.012] [nas] [error] Radio link failure detected
[2024-05-08 17:11:39.012] [nas] [info] UE switches to state [CM-IDLE]                                                                                                                                           [2024-05-08 17:11:39.012] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:39.012] [nas] [info] UE switches to state [MM-REGISTERED/PLMN-SEARCH]
[2024-05-08 17:11:39.012] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:40.139] [rrc] [debug] New signal detected for cell[3], total [1] cells in coverage                                                                                                            [2024-05-08 17:11:41.096] [nas] [error] PLMN selection failure, no cells in coverage
[2024-05-08 17:11:42.482] [rrc] [warning] Acceptable cell selection failed in [1] cells. [1] no SI, [0] reserved, [0] barred, ftai [0]                                                                          [2024-05-08 17:11:42.482] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2024-05-08 17:11:43.298] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:44.399] [nas] [info] UE switches to state [MM-REGISTERED/NO-CELL-AVAILABLE]                                                                                                                   [2024-05-08 17:11:49.539] [nas] [info] Selected plmn[208/93]
[2024-05-08 17:11:49.539] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]                                                                                                                     [2024-05-08 17:11:49.539] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:49.539] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-08 17:11:54.346] [rrc] [debug] Signal lost for cell[3], total [0] cells in coverage                                                                                                                    [2024-05-08 17:11:54.346] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:54.346] [nas] [info] UE switches to state [MM-REGISTERED/PLMN-SEARCH]
[2024-05-08 17:11:54.346] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:55.347] [rrc] [debug] New signal detected for cell[4], total [1] cells in coverage
[2024-05-08 17:11:56.510] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:58.712] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:59.540] [nas] [info] UE switches to state [MM-REGISTERED/NO-CELL-AVAILABLE]
[2024-05-08 17:11:59.541] [nas] [info] Selected plmn[208/93]                                                                                                                                                    [2024-05-08 17:11:59.541] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]                                                                                                                     [2024-05-08 17:11:59.541] [nas] [info] UE switches to state [MM-REGISTERED/PS]
[2024-05-08 17:11:59.541] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-08 17:12:08.216] [nas] [debug] Uplink data status changed PSI[1] pending[true]
[2024-05-08 17:12:08.216] [nas] [debug] Service request required due to [IDLE-UPLINK-DATA-PENDING]
[2024-05-08 17:12:08.216] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_data]
[2024-05-08 17:12:08.216] [nas] [debug] Sending Service Request due to [IDLE-UPLINK-DATA-PENDING]
[2024-05-08 17:12:08.216] [nas] [info] UE switches to state [MM-SERVICE-REQUEST-INITIATED]
[2024-05-08 17:12:08.216] [rrc] [debug] Sending RRC Setup Request
[2024-05-08 17:12:08.553] [rrc] [debug] Signal lost for cell[4], total [0] cells in coverage
[2024-05-08 17:12:08.553] [nas] [error] Active cell change in [CM-IDLE] state while MM specific procedure is ongoing
[2024-05-08 17:12:08.553] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-08 17:12:08.553] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-08 17:12:08.553] [nas] [error] PLMN selection failure, no cells in coverage
[2024-05-08 17:12:09.742] [rrc] [debug] New signal detected for cell[5], total [1] cells in coverage
[2024-05-08 17:12:10.823] [nas] [error] PLMN selection failure, no cells in coverage

💡 The reason of "signal lost" is a fake default gateways in GnB's deployment config:

      annotations:
        k8s.v1.cni.cncf.io/networks: >-
          [ { "name": "n2network-ueransim", "interface": "n2", "ips": [
          "10.100.50.250/29" ], "gateway": [ "10.100.50.254" ] }, { "name":
          "n3network-ueransim", "interface": "n3", "ips": [ "10.100.50.236/28"
          ], "gateway": [ "10.100.50.238" ] } ]

Solution is to remove , "gateway": [ "10.100.50.nnn" ] -- TODO: find a way how to make corrections to a helm deployment.

📝 Known bug: #465

Every peer should be configured to be monitored (see gtp_peer config option)

✔ fixed in #551

OK. See the difference in the first hop of traceroute from UE:

root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build# traceroute -i uesimtun0 www.google.com -w 1 -m3
traceroute to www.google.com (74.125.131.147), 3 hops max, 60 byte packets
 1  10.233.110.134 (10.233.110.134)  2.038 ms  1.972 ms  1.958 ms
 2  188-120-253-172.kube-prometheus-stack-kube-scheduler.kube-system.svc.cluster.local (188.120.253.172)  1.949 ms  1.941 ms  1.933 ms
 3  10.0.0.1 (10.0.0.1)  1.919 ms  1.910 ms  1.887 ms
root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build# traceroute -i uesimtun0 1.1.1.1 -w 1 -m3
traceroute to 1.1.1.1 (1.1.1.1), 3 hops max, 60 byte packets
 1  10.233.110.158 (10.233.110.158)  1.277 ms  1.263 ms  1.256 ms
 2  188-120-253-172.kubernetes.default.svc.cluster.local (188.120.253.172)  1.248 ms  1.239 ms  1.217 ms
 3  10.0.0.1 (10.0.0.1)  1.208 ms  1.197 ms  1.188 ms
root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build#

[PapaySail]

✔ Open5GS & Calico BGP - successfull

• tested by @pirog-spb

✔ Open5GS & Calico BGP with Slices - successfull

Running pods list

name	namespace	restarts	owners	node	qos	age	status
eupf-6c4768c9b4-7k45t	open5gs	0	ReplicaSet (eupf-6c4768c9b4)	edgecom-test-node01	Guaranteed	1715443181000	Running
eupf2-54d595fdd4-66n4x	open5gs	0	ReplicaSet (eupf2-54d595fdd4)	edgecom-test-node01	Guaranteed	1715443982000	Running
gnb-ueransim-gnb-585fdf8c5b-cmg4f	open5gs	0	ReplicaSet (gnb-ueransim-gnb-585fdf8c5b)	edgecom-test-node01	BestEffort	1715444893000	Running
open5gs-amf-6647bbf488-276sl	open5gs	0	ReplicaSet (open5gs-amf-6647bbf488)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-ausf-6b6c79777b-l94fb	open5gs	0	ReplicaSet (open5gs-ausf-6b6c79777b)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-bsf-584bbc4f74-pj9dl	open5gs	0	ReplicaSet (open5gs-bsf-584bbc4f74)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-mongodb-6b7cfd5c8d-sj9wg	open5gs	0	ReplicaSet (open5gs-mongodb-6b7cfd5c8d)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-nrf-c7dcc4c7d-qg6ll	open5gs	0	ReplicaSet (open5gs-nrf-c7dcc4c7d)	edgecom-test-node01	BestEffort	1715444456000	Running
open5gs-nssf-ff6c4d95b-7wz9x	open5gs	0	ReplicaSet (open5gs-nssf-ff6c4d95b)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-pcf-5c4846dd6b-47xbn	open5gs	2	ReplicaSet (open5gs-pcf-5c4846dd6b)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-populate-57cbd7d569-vr2pt	open5gs	0	ReplicaSet (open5gs-populate-57cbd7d569)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-smf-7df749f7-2tps5	open5gs	0	ReplicaSet (open5gs-smf-7df749f7)	edgecom-test-node01	BestEffort	1715444595000	Running
open5gs-smf2-6bf4d967c-sjwqg	open5gs	0	ReplicaSet (open5gs-smf2-6bf4d967c)	edgecom-test-node01	BestEffort	1715444801000	Running
open5gs-udm-5f8cfcd-bdmft	open5gs	0	ReplicaSet (open5gs-udm-5f8cfcd)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-udr-788f54787-xxjs9	open5gs	2	ReplicaSet (open5gs-udr-788f54787)	edgecom-test-node01	BestEffort	1715444455000	Running
open5gs-webui-699c9bf9d8-lwtqw	open5gs	0	ReplicaSet (open5gs-webui-699c9bf9d8)	edgecom-test-node01	BestEffort	1715444455000	Running
ueransim1-ueransim-ues-567d89c5dd-hc8p2	open5gs	0	ReplicaSet (ueransim1-ueransim-ues-567d89c5dd)	edgecom-test-node01	BestEffort	1715444957000	Running
ueransim2-ueransim-ues-79669647f6-wzc4f	open5gs	0	ReplicaSet (ueransim2-ueransim-ues-79669647f6)	edgecom-test-node01	BestEffort	1715445106000	Running

[PapaySail]

✔ Open5GS & Load Balanced eUPF - successfull

Details

PS C:\Users\SergO\Documents\GitHub\eupf\docs\deployments\open5gs-with-scaling-eupf> kubectl get po -n open5gs -l "app.kubernetes.io/name=eupf" -o wide
NAME     READY   STATUS    RESTARTS   AGE     IP               NODE                  NOMINATED NODE   READINESS GATES
eupf-0   1/1     Running   0          3m40s   10.233.110.136   edgecom-test-node01   <none>           <none>
eupf-1   1/1     Running   0          3m20s   10.233.110.132   edgecom-test-node01   <none>           <none>

SMF log:

05/13 16:35:22.646: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1010)                                       05/13 16:35:22.646: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3051)
05/13 16:35:22.647: [sbi] WARNING: Try to discover [nudm-sdm] (../lib/sbi/path.c:348)
05/13 16:35:22.648: [sbi] INFO: [1bafd4d6-1146-41ef-bd74-d3c6b28ef87a] (NF-discover) NF registered (../lib/sbi/nnrf-handler.c:829)
05/13 16:35:22.648: [sbi] INFO: [1bafd4d6-1146-41ef-bd74-d3c6b28ef87a] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:8505/13 16:35:22.653: [sbi] WARNING: Try to discover [npcf-smpolicycontrol] (../lib/sbi/path.c:348)                                    05/13 16:35:22.654: [sbi] INFO: [2b90c5d6-1146-41ef-9d4f-e5658e726d87] (NF-discover) NF registered (../lib/sbi/nnrf-handler.c:829)
05/13 16:35:22.654: [sbi] INFO: [2b90c5d6-1146-41ef-9d4f-e5658e726d87] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:85
05/13 16:35:22.661: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.2] IPv6[] (../src/smf/npcf-handler.c:528)   
05/13 16:35:22.663: [gtp] INFO: gtp_connect() [10.233.110.136]:2152 (../lib/gtp/path.c:60)

05/13 16:38:04.317: [smf] INFO: [Added] Number of SMF-UEs is now 2 (../src/smf/context.c:1010)
05/13 16:38:04.317: [smf] INFO: [Added] Number of SMF-Sessions is now 2 (../src/smf/context.c:3051)
05/13 16:38:04.325: [smf] INFO: UE SUPI[imsi-999700000000002] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/npcf-handler.c:528)
05/13 16:38:04.327: [gtp] INFO: gtp_connect() [10.233.110.132]:2152 (../lib/gtp/path.c:60)

UE1 console:

bash-5.1# ping -I uesimtun0 10.233.110.136
PING 10.233.110.136 (10.233.110.136): 56 data bytes
64 bytes from 10.233.110.136: seq=0 ttl=64 time=0.815 ms
64 bytes from 10.233.110.136: seq=1 ttl=64 time=0.858 ms
64 bytes from 10.233.110.136: seq=2 ttl=64 time=1.915 ms
64 bytes from 10.233.110.136: seq=3 ttl=64 time=1.381 ms
64 bytes from 10.233.110.136: seq=4 ttl=64 time=1.500 ms
^C
--- 10.233.110.136 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.815/1.293/1.915 ms
bash-5.1# ping -I uesimtun0 10.233.110.132
PING 10.233.110.132 (10.233.110.132): 56 data bytes
^C
--- 10.233.110.132 ping statistics ---
15 packets transmitted, 0 packets received, 100% packet loss

bash-5.1# ip a s dev uesimtun0
4: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.11.0.2/32 scope global uesimtun0

UE2 console:

bash-5.1# ping -I uesimtun0 10.233.110.136 -c 5
PING 10.233.110.136 (10.233.110.136): 56 data bytes

--- 10.233.110.136 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
bash-5.1# ping -I uesimtun0 10.233.110.132 -c 5
PING 10.233.110.132 (10.233.110.132): 56 data bytes
64 bytes from 10.233.110.132: seq=0 ttl=64 time=0.939 ms
64 bytes from 10.233.110.132: seq=1 ttl=64 time=1.139 ms
64 bytes from 10.233.110.132: seq=2 ttl=64 time=1.100 ms
64 bytes from 10.233.110.132: seq=3 ttl=64 time=6.400 ms
64 bytes from 10.233.110.132: seq=4 ttl=64 time=0.976 ms

--- 10.233.110.132 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.939/2.110/6.400 ms
bash-5.1# ip a s dev uesimtun0
4: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.11.0.3/32 scope global uesimtun0

[PapaySail]

✔ Open5GS & srsRAN & Calico BGP
Check steps

1. exec shell in UE pod -- wrong command. Correct will be: kubectl -n srs-open5gs exec -ti statefulset/srsran-srsran5gue-zmq-gnb -- /bin/bash
2. run ICMP test -- unsuccessfull

Details

PS C:\Users\SergO\Documents\GitHub\eupf\docs\deployments\srsran-gnb> kubectl -n srs-open5gs exec -ti statefulset/srsran-srsran5gue-zmq-gnb -- /bin/bash
Defaulted container "gnb" out of: gnb, ue
root@srsran-srsran5gue-zmq-gnb-0:/# ping -I tun_srsue 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.11.0.2 tun_srsue: 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2049ms

root@srsran-srsran5gue-zmq-gnb-0:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if409: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1430 qdisc noqueue state UP group default
    link/ether 66:fc:26:53:2b:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.233.110.137/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::64fc:26ff:fe53:2b14/64 scope link
       valid_lft forever preferred_lft forever
4: tun_srsue: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.11.0.2/24 scope global tun_srsue
       valid_lft forever preferred_lft forever
root@srsran-srsran5gue-zmq-gnb-0:/#
root@srsran-srsran5gue-zmq-gnb-0:/# ping -I tun_srsue 10.233.64.11
PING 10.233.64.11 (10.233.64.11) from 10.11.0.2 tun_srsue: 56(84) bytes of data.
^C
--- 10.233.64.11 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9224ms

root@srsran-srsran5gue-zmq-gnb-0:/#

SRSran GnB log:

+ /opt/srsRAN_Project/target/bin/gnb -c gnb.yml
Warning: Scheduling priority of thread "gnb_ctrl" not changed. Cause: Not enough privileges.
Warning: Scheduling priority of thread "du_cell#0" not changed. Cause: Not enough privileges.
Warning: Scheduling priority of thread "phy_worker" not changed. Cause: Not enough privileges.
2024-05-14T18:20:30.022340 [GNB     ] [W] Could not check scaling governor. filename=/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor error="No such file or directory"
2024-05-14T18:20:30.022387 [GNB     ] [W] DRM KMS polling is enabled, which may hinder performance. You can disable it using the "srsran_performance" script
2024-05-14T18:20:30.025034 [IO-EPOLL] [E] epoll_ctl failed for fd=0
2024-05-14T18:20:30.025035 [GNB     ] [E] Couldn't register stdin handler
2024-05-14T18:20:30.213949 [zmq:tx:0:0] [I] Binding to address tcp://127.0.0.1:2000.
2024-05-14T18:20:30.289730 [zmq:rx:0:0] [I] Connecting to address tcp://127.0.0.1:2001.
2024-05-14T18:20:35.823437 [MAC     ] [W] [    66.8] UL rnti=0x4601 lcid=61 CE: Discarding MAC CE. Cause: C-RNTI is not associated with any existing UE
2024-05-14T18:22:13.317015 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:14.341467 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:15.351123 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:18.415462 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001

OK. Solution is to change image to eupf:main
🆒 fact: after restart of upf, smf processes this events and sends command to create session as it was. So, UE gets connectivity immediately.

Details: PFCP restoration

eUPF log after restart:

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:    export GIN_MODE=release
 - using code:    gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /metrics                  --> github.com/edgecomllc/eupf/cmd/api/rest.(*ApiHandler).InitMetricsRoute.func1.1 (4 handlers)
2024/05/31 17:41:21 INF running on :8080
2024/05/31 17:41:21 INF running on :9090
2024/05/31 17:41:22 INF Got Association Setup Request from: 10.233.110.143.

2024/05/31 17:41:22 INF
Association Setup Request:
  Node ID: 10.233.110.143
  Recovery Time: 2024-05-30 08:13:36 +0000 UTC

2024/05/31 17:41:22 INF Saving new association: &{ID:10.233.110.143 Addr:10.233.110.143 NextSessionID:1 NextSequenceID:1 Sessions:map[] HeartbeatChannel:0xc0000ae4e0 FailedHeartbeats:0 HeartbeatsActive:false Mutex:{state:0 sema:0}}
2024/05/31 17:41:22 INF Got Session Establishment Request from: 10.233.110.143.
2024/05/31 17:41:22 INF
Session Establishment Request:
  CreatePDR ID: 1
    FAR ID: 1
    QER ID: 1
    URR ID: 1
    Source Interface: 1
    UE IPv4 Address: 10.11.0.3
  CreatePDR ID: 2
    Outer Header Removal: 0
    FAR ID: 2
    QER ID: 1
    Source Interface: 0
    TEID: 5034
    Ipv4: 10.233.64.11
    Ipv6: <nil>
    UE IPv4 Address: 10.11.0.3
  CreatePDR ID: 3
    Outer Header Removal: 0
    FAR ID: 1
    QER ID: 1
    Source Interface: 3
    TEID: 8912
    Ipv4: 10.233.64.11
    Ipv6: <nil>
  CreatePDR ID: 4
    Outer Header Removal: 0
    FAR ID: 3
    Source Interface: 0
    TEID: 5034
    Ipv4: 10.233.64.11
    Ipv6: <nil>
    SDF Filter: permit out 58 from ff02::2/128 to assigned
  CreateFAR ID: 1
    Apply Action: [2 0]
    Forwarding Parameters:
      Network Instance: internet
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.233.110.176 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
  CreateFAR ID: 2
    Apply Action: [2 0]
    Forwarding Parameters:
      Network Instance: internet
  CreateFAR ID: 3
    Apply Action: [2 0]
    Forwarding Parameters:
      Network Instance: internet
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:2 IPv4Address:10.233.110.143 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
  CreateQER ID: 1
    Gate Status DL: 0
    Gate Status UL: 0
    Max Bitrate DL: 999999
    Max Bitrate UL: 999999
    QFI: 1
  CreateURR ID: 1
    Measurement Method: 2
    Volume Threshold: &{Flags:1 TotalVolume:104857600 UplinkVolume:0 DownlinkVolume:0}
  CreateBAR ID: 1

2024/05/31 17:41:22 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:2960058634 LocalIP:188803338 TransportLevelMarking:0}
2024/05/31 17:41:22 INF WARN: No OuterHeaderCreation
2024/05/31 17:41:22 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:188803338 TransportLevelMarking:0}
2024/05/31 17:41:22 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:2406410506 LocalIP:188803338 TransportLevelMarking:0}
2024/05/31 17:41:22 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:999999000 MaxBitrateDL:999999000 StartUL:0 StartDL:0}
2024/05/31 17:41:22 Matched groups: ["permit out 58 from ff02::2/128 to assigned" "58" "ff02::2" "128" "" "assigned" "" ""]
2024/05/31 17:41:22 INF Session Establishment Request from 10.233.110.143 accepted.

SMF log:

05/31 16:28:02.911: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1010)
05/31 16:28:02.911: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3051)
05/31 16:28:02.919: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/npcf-handler.c:528)
05/31 17:17:05.321: [sbi] INFO: [da4a3af0-15b8-41ef-8e58-3d6de1144d1a] NF expired (../lib/sbi/nf-sm.c:318)
05/31 17:17:05.326: [sbi] INFO: [e9c93eb8-15b8-41ef-aacb-15e80733941f] NF expired (../lib/sbi/nf-sm.c:318)
05/31 17:17:05.374: [sbi] INFO: [34d464ce-1e5a-41ef-9bcf-bd90bf5b1752] NF expired (../lib/sbi/nf-sm.c:318)
05/31 17:41:22.721: [pfcp] ERROR: Remote PFCP restarted [3926161136<3926166081] in Heartbeat RSP (../lib/pfcp/handler.c:76)
05/31 17:41:22.721: [smf] INFO: PFCP de-associated [10.233.64.11]:8805 (../src/smf/pfcp-sm.c:201)
05/31 17:41:22.722: [pfcp] WARNING: F-TEID allocation/release not supported with peer [10.233.64.11]:8805 (../lib/pfcp/handler.c:182)
05/31 17:41:22.722: [smf] INFO: PFCP associated [10.233.64.11]:8805 (../src/smf/pfcp-sm.c:186)
05/31 17:41:22.722: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/pfcp-sm.c:442)
05/31 17:41:22.722: [smf] ERROR: PFCP restoration (../src/smf/pfcp-sm.c:197)

Fixed in #552

[PapaySail]

Free5GC & Calico BGP

4. install eupf make upf :: problem: hardcoded "master": "eth0" in values/eupf.yaml

Pod can't create. Status: ContainerCreating

Pod description:

  Warning  FailedCreatePodSandBox  74s (x16 over 103s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc err
or: code = Unknown desc = failed to setup network for sandbox "7b793c609a914881a32db2f7e3350992cfd1162b17790a985e4e62b6f841f864": plugin type="m
ultus" name="multus-cni-network" failed (add): [free5gc24/eupf-57ccfb74bd-m4js5:n3network-free5gc-free5gc-upf]: error adding container to networ
k "n3network-free5gc-free5gc-upf": failed to lookup master "eth0": Link not found

1. exec shell in UE pod :: error in doc : correct command is:
   kubectl -n free5gc exec -ti deployment/ueransim1-ue -- /bin/bash

✔ fixed in #551

[PapaySail]

✔ Docker-compose Free5GC

docker-compose up -d

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose up -d
[+] Running 21/21
 _ Network free5gc-compose_privnet_n6  Created                                                                                                      0.1s
 _ Network free5gc-compose_default     Created                                                                                                      0.1s
 _ Network free5gc-compose_privnet     Created                                                                                                      0.1s
 _ Container prometheus                Started                                                                                                      1.3s
 _ Container upf                       Started                                                                                                      1.6s
 _ Container grafana                   Started                                                                                                      1.7s
 _ Container eupf                      Started                                                                                                      2.6s
 _ Container nat                       Started                                                                                                      2.4s
 _ Container mongodb                   Started                                                                                                      0.9s
 _ Container ui                        Started                                                                                                      1.3s
 _ Container webui                     Started                                                                                                      2.1s
 _ Container nrf                       Started                                                                                                      2.4s
 _ Container nssf                      Started                                                                                                      3.9s
 _ Container udm                       Started                                                                                                      3.9s
 _ Container ausf                      Started                                                                                                      3.9s
 _ Container amf                       Started                                                                                                      3.4s
 _ Container pcf                       Started                                                                                                      4.1s
 _ Container smf                       Started                                                                                                      4.0s
 _ Container udr                       Started                                                                                                      3.9s
 _ Container ueransim                  Started                                                                                                      4.2s
 _ Container n3iwf                     Started                                                                                                      4.5s
sergo@edgecom:~/edgecom24/free5gc-compose$

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec  ueransim bash
root@1d4b478b1c91:/ueransim# ./nr-ue -c config/uecfg.yaml
UERANSIM v3.2.6
[2024-05-16 17:37:49.842] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-16 17:37:49.843] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2024-05-16 17:37:49.846] [nas] [info] Selected plmn[208/93]
[2024-05-16 17:37:52.343] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2024-05-16 17:37:52.343] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2024-05-16 17:37:52.343] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 17:37:52.343] [nas] [debug] Sending Initial Registration
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2024-05-16 17:37:52.343] [rrc] [debug] Sending RRC Setup Request
[2024-05-16 17:37:52.346] [rrc] [info] RRC connection established
[2024-05-16 17:37:52.346] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2024-05-16 17:37:52.346] [nas] [info] UE switches to state [CM-CONNECTED]
[2024-05-16 17:37:52.378] [nas] [debug] Authentication Request received
[2024-05-16 17:37:52.389] [nas] [debug] Security Mode Command received
[2024-05-16 17:37:52.389] [nas] [debug] Selected integrity[2] ciphering[0]
[2024-05-16 17:37:52.440] [nas] [debug] Registration accept received
[2024-05-16 17:37:52.440] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-16 17:37:52.440] [nas] [debug] Sending Registration Complete
[2024-05-16 17:37:52.440] [nas] [info] Initial Registration is successful
[2024-05-16 17:37:52.440] [nas] [debug] Sending PDU Session Establishment Request
[2024-05-16 17:37:52.463] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 17:37:52.709] [nas] [debug] PDU Session Establishment Accept received
[2024-05-16 17:37:52.709] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-16 17:37:52.725] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.60.0.1] is up.

### at another console
sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec  ueransim bash
root@1d4b478b1c91:/ueransim# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.60.0.1/32 scope global uesimtun0
       valid_lft forever preferred_lft forever
689: eth0@if690: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:0a:64:c8:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.100.200.16/24 brd 10.100.200.255 scope global eth0
       valid_lft forever preferred_lft forever
root@1d4b478b1c91:/ueransim#
root@1d4b478b1c91:/ueransim# ping -I uesimtun0 1.1.1.1

PING 1.1.1.1 (1.1.1.1) from 10.60.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.19 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.58 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.68 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=55 time=1.66 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=55 time=1.59 ms
^C
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 1.584/1.738/2.190/0.228 ms
root@1d4b478b1c91:/ueransim#

eupf logs:
sudo docker-compose logs edgecom-upf |less

eupf  | 2024/05/16 17:37:52 Handling PFCP message from 10.100.200.15:8805
eupf  | 2024/05/16 17:37:52 Got Session Modification Request from: 10.100.200.15.
eupf  | 2024/05/16 17:37:52 Finding association for 10.100.200.15
eupf  | 2024/05/16 17:37:52 Finding session 2
eupf  | 2024/05/16 17:37:52
eupf  | Session Modification Request:
eupf  |   UpdatePDR ID: 2
eupf  |     FAR ID: 2
eupf  |     Source Interface: 1
eupf  |     UE IPv4 Address: 10.60.0.1
eupf  |   UpdateFAR ID: 2
eupf  |     Apply Action: [2]
eupf  |     Update forwarding Parameters:
eupf  |       Network Instance: internet
eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
eupf  | 2024/05/16 17:37:52 Updating FAR info: 2, {FarInfo:{Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMark
ing:0} GlobalId:1}
eupf  | 2024/05/16 17:37:52 EBPF: Update FAR: internalId=1, farInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 Transport
LevelMarking:0}
eupf  | 2024/05/16 17:37:52 Both F-TEID IE and UE IP Address IE are missing
eupf  | 2024/05/16 17:37:57 Sent Heartbeat Request to: 10.100.200.15
eupf  | 2024/05/16 17:37:57 Received 16 bytes from 10.100.200.15:8805

✔ Free5GC with UpLink Classifier config throught three eUPFs

docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d

sergo@edgecom:~/edgecom24/free5gc-compose$ git checkout ulcl-n9upf-experimetns
Branch 'ulcl-n9upf-experimetns' set up to track remote branch 'ulcl-n9upf-experimetns' from 'origin'.
Switched to a new branch 'ulcl-n9upf-experimetns'

docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose ps
NAME                IMAGE                           COMMAND                  SERVICE             CREATED             STATUS              PORTS
amf                 free5gc/amf:v3.3.0              "./amf -c ./config/a_"   free5gc-amf         2 minutes ago       Up 2 minutes        8000/tcp
ausf                free5gc/ausf:v3.3.0             "./ausf -c ./config/_"   free5gc-ausf        2 minutes ago       Up 2 minutes        8000/tcp
grafana             grafana/grafana                 "/run.sh"                grafana             2 minutes ago       Up 2 minutes        0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
i-eupf              ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-i-upf       2 minutes ago       Up 2 minutes        0.0.0.0:8082->8080/tcp, :::8082->8080/tcp, 0.0.0.0:9092->9090/tcp, :::9092->9090/tcp
mongodb             mongo                           "docker-entrypoint.s_"   db                  2 minutes ago       Up 2 minutes        27017/tcp
n3iwf               free5gc/n3iwf:v3.3.0            "sh -c './n3iwf-ipse_"   free5gc-n3iwf       2 minutes ago       Up 2 minutes
nat                 free5gc-compose-edgecom-nat     "sh /app/bin/busy-po_"   edgecom-nat         2 minutes ago       Up 2 minutes
nrf                 free5gc/nrf:v3.3.0              "./nrf -c ./config/n_"   free5gc-nrf         2 minutes ago       Up 2 minutes        8000/tcp
nssf                free5gc/nssf:v3.3.0             "./nssf -c ./config/_"   free5gc-nssf        2 minutes ago       Up 2 minutes        8000/tcp
pcf                 free5gc/pcf:v3.3.0              "./pcf -c ./config/p_"   free5gc-pcf         2 minutes ago       Up 2 minutes        8000/tcp
prometheus          prom/prometheus                 "/bin/prometheus --c_"   prometheus          2 minutes ago       Up 2 minutes        0.0.0.0:9091->9090/tcp, :::9091->9090/tcp
psa-eupf            ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-psa-upf     2 minutes ago       Up 2 minutes        0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp
psa-eupf-2          ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-psa-upf-2   2 minutes ago       Up 2 minutes
smf                 free5gc/smf:v3.3.0              "./smf -c ./config/s_"   free5gc-smf         2 minutes ago       Up 2 minutes        8000/tcp
udm                 free5gc/udm:v3.3.0              "./udm -c ./config/u_"   free5gc-udm         2 minutes ago       Up 2 minutes        8000/tcp
udr                 free5gc/udr:v3.3.0              "./udr -c ./config/u_"   free5gc-udr         2 minutes ago       Up 2 minutes        8000/tcp
ueransim            free5gc/ueransim:v3.3.0         "./nr-gnb -c ./confi_"   ueransim            2 minutes ago       Up 2 minutes
webui               free5gc/webui:v3.3.0            "./webui -c ./config_"   free5gc-webui       2 minutes ago       Up 2 minutes        0.0.0.0:5000->5000/tcp, :::5000->5000/tcp
sergo@edgecom:~/edgecom24/free5gc-compose$
sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash
root@aaad991d992b:/ueransim# ./nr-ue -c config/uecfg.yaml &
[1] 24
root@aaad991d992b:/ueransim# UERANSIM v3.2.6
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-16 18:01:03.356] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2024-05-16 18:01:03.356] [nas] [info] Selected plmn[208/93]
[2024-05-16 18:01:03.356] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2024-05-16 18:01:03.356] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2024-05-16 18:01:03.357] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 18:01:03.357] [nas] [debug] Sending Initial Registration
[2024-05-16 18:01:03.357] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2024-05-16 18:01:03.357] [rrc] [debug] Sending RRC Setup Request
[2024-05-16 18:01:03.357] [rrc] [info] RRC connection established
[2024-05-16 18:01:03.357] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2024-05-16 18:01:03.357] [nas] [info] UE switches to state [CM-CONNECTED]
[2024-05-16 18:01:03.387] [nas] [debug] Authentication Request received
[2024-05-16 18:01:03.398] [nas] [debug] Security Mode Command received
[2024-05-16 18:01:03.398] [nas] [debug] Selected integrity[2] ciphering[0]
[2024-05-16 18:01:03.451] [nas] [debug] Registration accept received
[2024-05-16 18:01:03.451] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-16 18:01:03.451] [nas] [debug] Sending Registration Complete
[2024-05-16 18:01:03.451] [nas] [info] Initial Registration is successful
[2024-05-16 18:01:03.451] [nas] [debug] Sending PDU Session Establishment Request
[2024-05-16 18:01:03.451] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 18:01:03.720] [nas] [debug] PDU Session Establishment Accept received
[2024-05-16 18:01:03.720] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-16 18:01:03.735] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.62.0.1] is up.

root@aaad991d992b:/ueransim# ping -I uesimtun0 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.62.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.01 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.61 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.66 ms
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.608/1.759/2.007/0.176 ms
root@aaad991d992b:/ueransim# ping -I uesimtun0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.62.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=16.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=17.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=17.0 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.873/16.951/17.002/0.056 ms

Logs

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-i-upf |less -R
look for 8.8.8.8

i-eupf  | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17.
i-eupf  |
i-eupf  |
i-eupf  | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17
i-eupf  | 2024/05/16 18:01:03 INF Finding session 2
i-eupf  | 2024/05/16 18:01:03 INF
i-eupf  | Session Modification Request:
i-eupf  |   CreatePDR ID: 3
i-eupf  |     Outer Header Removal: 0
i-eupf  |     FAR ID: 3
i-eupf  |     QER ID: 1
i-eupf  |     Source Interface: 0
i-eupf  |     TEID: 1
i-eupf  |     Ipv4: 10.100.200.14
i-eupf  |     Ipv6: <nil>
i-eupf  |     UE IPv4 Address: 10.62.0.1
i-eupf  |     SDF Filter: permit out ip from 10.62.0.1 to 8.8.8.8/32
i-eupf  |   CreatePDR ID: 4
i-eupf  |     Outer Header Removal: 0
i-eupf  |     FAR ID: 4
i-eupf  |     QER ID: 1
i-eupf  |     Source Interface: 1
i-eupf  |     TEID: 4
i-eupf  |     Ipv4: 10.100.200.14
i-eupf  |     Ipv6: <nil>
i-eupf  |   CreateFAR ID: 3
i-eupf  |     Apply Action: [2]
i-eupf  |     Forwarding Parameters:
i-eupf  |       Network Instance: internet
i-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
i-eupf  |   CreateFAR ID: 4
i-eupf  |     Apply Action: [2]
i-eupf  |     Forwarding Parameters:
i-eupf  |       Network Instance: internet
i-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.19 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
i-eupf  |
i-eupf  |
i-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=2, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 4, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=3, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:2 QerId:1 SdfFilter:{Protocol:1 SrcAddress:{Type:1 Ip:10.62.0.1 Mask:ffffffff} SrcPortRange:{LowerBound:0 UpperBound:65535} DstAddress:{Type:1 Ip:8.8.8.8 Mask:ffffffff} DstPortRange:{LowerBound:0 UpperBound:65535}}}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=4, pdrInfo={OuterHeaderRemoval:0 FarId:3 QerId:1 SdfFilter:<nil>}
i-eupf  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf |less -R

psa-eupf  | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17.
psa-eupf  | 2024/05/16 18:01:03 INF
psa-eupf  | Session Establishment Request:
psa-eupf  |   CreatePDR ID: 1
psa-eupf  |     Outer Header Removal: 0
psa-eupf  |     FAR ID: 1
psa-eupf  |     QER ID: 2
psa-eupf  |     QER ID: 1
psa-eupf  |     Source Interface: 0
psa-eupf  |     TEID: 1
psa-eupf  |     Ipv4: 10.100.200.15
psa-eupf  |     Ipv6: <nil>
psa-eupf  |     UE IPv4 Address: 10.62.0.1
psa-eupf  |   CreatePDR ID: 2
psa-eupf  |     FAR ID: 2
psa-eupf  |     QER ID: 2
psa-eupf  |     QER ID: 1
psa-eupf  |     Source Interface: 1
psa-eupf  |     UE IPv4 Address: 10.62.0.1
psa-eupf  |   CreateFAR ID: 1
psa-eupf  |     Apply Action: [2]
psa-eupf  |     Forwarding Parameters:
psa-eupf  |       Network Instance: internet
psa-eupf  |   CreateFAR ID: 2
psa-eupf  |     Apply Action: [2]
psa-eupf  |     Forwarding Parameters:
psa-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:2 IPv4Address:10.100.200.14 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
psa-eupf  |   CreateQER ID: 2
psa-eupf  |     Gate Status DL: 0
psa-eupf  |     Gate Status UL: 0
psa-eupf  |     QFI: 1
psa-eupf  |   CreateQER ID: 1
psa-eupf  |     Gate Status DL: 0
psa-eupf  |     Gate Status UL: 0
psa-eupf  |     Max Bitrate DL: 1000000
psa-eupf  |     Max Bitrate UL: 2000000
psa-eupf  |     QFI: 0
psa-eupf  |
psa-eupf  |
psa-eupf  | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation
psa-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving QER info to session: 2, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=1, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:<nil>}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:<nil>}
psa-eupf  | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted.
psa-eupf  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf-2 |less -R

psa-eupf-2  | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17.
psa-eupf-2  | 2024/05/16 18:01:03 INF
psa-eupf-2  | Session Establishment Request:
psa-eupf-2  |   CreatePDR ID: 1
psa-eupf-2  |     Outer Header Removal: 0
psa-eupf-2  |     FAR ID: 1
psa-eupf-2  |     QER ID: 1
psa-eupf-2  |     Source Interface: 0
psa-eupf-2  |     TEID: 1
psa-eupf-2  |     Ipv4: 10.100.200.16
psa-eupf-2  |     Ipv6: <nil>
psa-eupf-2  |     UE IPv4 Address: 10.62.0.1
psa-eupf-2  |   CreateFAR ID: 1
psa-eupf-2  |     Apply Action: [2]
psa-eupf-2  |     Forwarding Parameters:
psa-eupf-2  |       Network Instance: internet
psa-eupf-2  |   CreateQER ID: 1
psa-eupf-2  |     Gate Status DL: 0
psa-eupf-2  |     Gate Status UL: 0
psa-eupf-2  |     Max Bitrate DL: 1000000
psa-eupf-2  |     Max Bitrate UL: 2000000
psa-eupf-2  |     QFI: 0
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:<nil>}
psa-eupf-2  | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted.
psa-eupf-2  | 2024/05/16 18:01:03 INF Received 137 bytes from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17.
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17
psa-eupf-2  | 2024/05/16 18:01:03 INF Finding session 2
psa-eupf-2  | 2024/05/16 18:01:03 INF
psa-eupf-2  | Session Modification Request:
psa-eupf-2  |   CreatePDR ID: 2
psa-eupf-2  |     FAR ID: 2
psa-eupf-2  |     QER ID: 1
psa-eupf-2  |     Source Interface: 1
psa-eupf-2  |     UE IPv4 Address: 10.62.0.1
psa-eupf-2  |   CreateFAR ID: 2
psa-eupf-2  |     Apply Action: [2]
psa-eupf-2  |     Forwarding Parameters:
psa-eupf-2  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:4 IPv4Address:10.100.200.14 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:<nil>}
psa-eupf-2  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:04 INF Handling PFCP message from 10.100.200.17:8805