omnet-bench

Implements a benchmark environment using OMNeT++.

For the experiments conducted in the paper, please checkout to the experiments branch: https://github.com/network-digital-twin/omnet-bench/tree/experiments

Introduction

Essentially, the implementation lies within the src/zte_qos/ folder, which is an OMNeT++ project taking INET from src/inet/ as a reference project. The INET project is a git submodule in this repository.

Inside the main zte_qos project, there are three important folders:

1. src/: place to hold the source code of:

   1. networks (e.g., TestNet);
   2. network components like the terminal, switch, link and packet;
   3. QoS modules;
   4. other utilities (e.g., sink, JSON handler by nlohmann/json).

   This folder also contains two executables pre-built by running the Makefile inside.

2. simulations/: place to hold the .ini files containing the configurations of:

   1. execution dependencies;
   2. network-specific Config profiles;
   3. parameter values of the QoS modules.

   The benchmark will be executed from this folder, using the executable in src/ as well as the omnetpp.ini configuration file.

3. tools/: place to hold two important Python scripts:

   1. topology generator: generates the xxNet.ned file, which represents the network topology as well as the locations of the switch info data and the trace data. For networks with customized names, extra Config blocks will be provided by the script for the users to add to the corresponding .ini configuration file. When utilizing JSON info files, these JSON files will be transformed from the original YAML info files.
   2. result analyzer: extracts result logs from a .log file generated by the simulation, dumps packet results to a *-res-packets.csv file, and processes the simulation metrics into a *-res-simulation.json file. Per-flow packet results are processed into a *-res-flows.csv file.

Getting Started

This section clarifies the steps needed to run the benchmark without building the OMNeT++ project.

Requirements

To run the benchmark, the OMNeT++ environment is mandatory since the INET project has to be rebuilt. To install OMNeT++, refer to the official installation guide.

Furthermore, a Python environment is needed to execute the topology generation script and the result analysis script.

The Python dependencies are listed as follows:

1. PyYAML==6.0.1: loading/dumping of YAML files, required by the topology generator;
2. pandas==2.2.0: loading/dumping of CSV files, required by the result analyzer;
3. pyarrow==15.0.0: future dependency of pandas.

Clarifications

Below specifies the minimum operations required to perform a benchmark on a specified network with a specified packet trace.

NOTE: to reduce the number of required operations, many settings are kept as default values. Specifically:

1. all switch info YAML files should be placed in one single folder;
2. the network will be named as "ZteNet", and generated in src/zte_qos/src/networks/gen/;
3. results will be recorded to src/zte_qos/simulations/results/.
4. JSON parsing instead of YAML parsing will be enabled.

1. Prepare network & trace data

Two categories of data are required:

1. info/<SWITCH_ID>.yaml: a directory, not necessarily named info/, should contain all the switch info YAML files. These files include the routing information of the switches, which is required by the RouteSelectors.
2. trace.txt: a text file specifying a trace of packets to be simulated on the network, which is required by the Terminal. Each line in the file represents a packet, whose attributes are separated by one single space. Lines starting with a "#" are ignored.

2. Generate network topology

Go to folder where the network topology generator resides:

cd src/zte_qos/tools/topo_gen/

Install the Python dependencies (remember to switch to a conda environment if conda is utilized):

python -m pip install -r requirements.txt

Then, run main.py as follow:

python main.py -i <INFO_DIR> -t <TRACE_FN> -j 1 -w <CORES>

where <INFO_DIR> is the folder path holding the switch info files, and <TRACE_FN> is the file path of the packet trace. -j enables JSON parsing of the info files. -w specifies the number of workers/cores used to load the switches info files in parallel. As an example, one can run:

python main.py -i ../../simulations/dataset/test/info/ -t ../../simulations/dataset/test/trace.txt -j 1 -w 36

The output will be:

100%|██████████| 3/3 [00:00<00:00, 15.78it/s]
"../../src/networks/gen/ZteNet.ned" successfully generated.

According to the output, the corresponding ZteNet.ned file has been successfully generated.

3. Run the simulation

After successful generation, go to the first src/ directory of the repository:

# should be at src/
cd ../../../

Then, run the simulation using the Makefile:

NET=ZteNet SIM_TIME=<SIM_TIME>s make run

where <SIM_TIME> specifies the simulation time duration in seconds. Note that this is the virtual simulation time, but not the real-world wall clock time. As an example, to simulate 60 seconds (the simulation stops when virtual time reaches 60s), one can run:

NET=ZteNet SIM_TIME=60s make run

The output will be:

git submodule update --init --recursive
cd zte_qos/simulations; ../src/zte_qos -m -u Cmdenv -c ZteNet omnetpp.ini --sim-time-limit=60s
OMNeT++ Discrete Event Simulation  (C) 1992-2022 Andras Varga, OpenSim Ltd.
Version: 6.0.2, build: 231006-7d5ca33dba, edition: Academic Public License -- NOT FOR COMMERCIAL USE
See the license for distribution terms and warranty disclaimer

Setting up Cmdenv...

Loading NED files from **/omnet-bench/src/zte_qos/src:  10
Loading NED files from **/omnet-bench/src/zte_qos/simulations:  1
Loading NED files from **/omnet-bench/src/inet/examples:  179
Loading NED files from **/omnet-bench/src/inet/showcases:  70
Loading NED files from **/omnet-bench/src/inet/src:  1142
Loading NED files from **/omnet-bench/src/inet/tests/validation:  5
Loading NED files from **/omnet-bench/src/inet/tests/networks:  6
Loading NED files from **/omnet-bench/src/inet/tutorials:  20

Preparing for running configuration ZteNet, run #0...
Redirecting output to file "**/omnet-bench/src/zte_qos/simulations/results/ZteNet-#0.log"...

End.

After the simulation, a .log file will be generated in the zte_qos/simulations/results/ folder. For this section, the name will always be ZteNet-#0.log, where #0 represents Run Number 0.

4. Analyze the simulation result

Go to the folder where the result analyzer resides:

cd zte_qos/tools/analysis/

Install the Python dependencies:

python -m pip install -r requirements.txt

Then, run main.py as follow:

python main.py -l <LOG_FN>

where <LOG_FN> is the file path of the result .log file. Again, as an example, one can run:

python main.py -l ../../simulations/results/ZteNet-#0.log

The output will be:

{
  "start": "2024-02-26 19:59:39.410424",
  "end": "2024-02-26 19:59:39.411428",
  "elapsed": 0.0010043,
  "num_events": 134,
  "num_packets": 5,
  "delay": {
    "min": 0.0,
    "max": 48000009999.99999,
    "avg": 21750007999.999996
  },
  "jitter": 17290896072.78975,
  "drop_rate": 0.0,
  "eps": 133426.26705167777
}
Successfully generated "..\..\simulations\results\TestNet-#0-res-simulation.json"
Successfully generated "..\..\simulations\results\TestNet-#0-res-packets.csv"
Successfully generated "..\..\simulations\results\TestNet-#0-res-flows.csv"

According to the output, the corresponding ZteNet-#0-res-simulation.json, ZteNet-#0-res-packets.csv, and ZteNet-#0-res-flows.csv files have been successfully generated. The output JSON is identical to the *-res-simulation.json file.

Below shows the resulted ZteNet-#0-res-packets.csv file for reference:

pid,src,dst,start_ts,end_ts,drop,module
150,0,0,2500000000.0,2500000000.0,0,TestNet.s_0.dstSink
135,0,2,2000000000.0,8250010000.0,0,TestNet.s_2.dstSink
123,0,1,1500000000.0,24750010000.0,0,TestNet.s_1.dstSink
33,0,1,1000000000.0,32250009999.999996,0,TestNet.s_1.dstSink
6,0,1,500000000.0,48500009999.99999,0,TestNet.s_1.dstSink

Below shows the resulted ZteNet-#0-res-flows.csv file for reference:

0,0,1,0.0,0.0,0.0
0,1,3,34166676666.66666,10312478956.207481,0.0
0,2,1,6250010000.0,0.0,0.0

The columns are ['src', 'dst', 'num_packets', 'delay', 'jitter', 'drop_rate'].

Speed Up Project Building

To enable multi-core building, set the temporary environment variable CORE to an integer and run as:

# <ROOT>/src/
CORE=16 make build

The same applies to other make commands.

Using a Customized Network

To specify a network with a different name from the default ZteNet, specify the network name in the network topology generator:

# <ROOT>/src/zte_qos/tools/topo-gen/
python main.py -i <INFO_DIR> -t <TRACE_FN> -n <NETWORK_NAME> -j 1 -w <CORES>

For example, to generate a network named NaNet, run as:

# <ROOT>/src/zte_qos/tools/topo-gen/
python main.py -i ../../simulations/dataset/test/info/ -t ../../simulations/dataset/test/trace.txt -n NaNet -j 1 -w 36

The output will suggest adding a Config block to an INI file:

100%|██████████| 3/3 [00:00<00:00, 15.78it/s]
"../../src/networks/gen/NaNet.ned" successfully generated.
Please add the following Config block to "../../simulations/config/GenNets.ini", if not exist:
```
[Config NaNet]
description = "NaNet."
network = zte_qos.networks.gen.NaNet
#sim-time-limit = 60s
#*.traceFile = "<TRACE_FN>"
###########################
###         QOS         ###
###########################
# Add QoS configurations here.
```

Add the Config block by coping and pasting the text between the code block marks to GenNets.ini. Then, run the simulation by changing the network name:

# <ROOT>/src/
NET=NaNet SIM_TIME=60s make run

A LOG file with a different name (NaNet-#0.log) will be generated in the result folder. Finally, use the result analyzer to specify this log file for the final results.

For Developers

If you modify the project C++ code or building configurations like include/linking, you need to generate the Makefiles and build the project again:

# <ROOT>/src/
make gen-makefiles
make build

If you modify only NED files and INI files, re-building is not required and you can run the simulation directly.