FuStIP - Full Stack IO Profiling

This tool aims to provide full stack profiling of IO on a system, mainly build up by bpf - therefore supporting Linux Kernel tracing. pidstat is used for syswide metrics, like cpu/memory/disk usage, for somewhat of a baseline of a system profile. For bpf, the tool uses two modes:

• summary: using bpftrace to build and output a summary IO profile, based on the tools/functionality the tool provides. This mode will output the aggregated results at the end of the script.
• detailed: using c wrapper scripts, providing deeper insights via a full per-event trace with nanosecond timestamps (Parquet), from which time-series and aggregate stats are derived. This mode also allows for generating visualizations of the data gathered.

Both modes filter along a nested-AND hierarchy — container ⊇ device ⊇ {comm, pid}:

• container (top): restrict to these containers (resolved to mount namespaces at runtime). Forces detailed mode — combining it with mode: summary is a pre-run error.
• device: narrow to these block devices (nvme and block layers). The nvme layer is single-device in summary mode (a multi-device filter is rejected); use detailed mode for multi-device.
• comm (command name, substring) and pid (tgid, exact): the process tier, unioned. With no container/device set, they collect system-wide.

Each unset tier is a pass-through; at least one filter is required per eBPF layer.

Repo Structure

This repo/tool is structured into some different sub directories, briefly described here:

FuStIP
|-  layers/                 (Supported Layers for IO Profiling)
|   |-  block/              (Block layer)
|   |-  fs/                 (File System layer)
|   |-  nvme/               (NVMe layer)
|   |-  sysstat/            (syswide metrics, cpu/mem/disk)
|- tests/                   (Tests/Workloads for different Layers)
|   |-  block_nvme/         (block and nvme layer)
|   |-  filesystem/         (fs and sysstat layer)
|   |-  nvme_direct/        (nvme layer over io_uring_cmd passthrough)
|   |-  sysstat/            (sysstat layer's cgroup memory accounting)
|   |-  placement/          (collector pinning; no root/device needed)
|- util/                    (shared utility/scripts for tool)
|- results/                 (results dir, generated by tool)
|- devbox.json              (devbox file, see prerequisites)

Prerequisites

This tool provides a devbox.json file, declaring packages+versions needed for using the tool. Please, refer to DevBox Docs for details on how to install and use, but follow this for a brief guide:

1. Install with curl -fsSL https://get.jetify.com/devbox | bash.
2. Create a .env file next to the provided devbox.json file, which can simply be empty, for overwriting/setting env vars.
3. Run command devbox run update_pip the first time you are setting up the tool.
4. Use devbox shell anywhere within this repo to enter the DevBox shell, which will include all packages required to use the tool. This is also what you should use for running/using the tool post-setup.

You can also use the tool without DevBox, in which case the packages specified in the devbox.json file are still needed, but you will have to ensure the correct versions etc. yourself. It is highly recommended to use DevBox, using Nix package manager under the hood, as it will provide everything needed, and this is how it's been tested.

Test Suite

The tool aims to ensure stability and correctness through a Flexible I/O Tester (fio) test suite, providing different workloads for validating the tool under both low and high load. Please, refer to the test dir for details on this, and note that it is divided into fs+syswide and block+nvme workload, basically differentiating between syscalls and requests going through the kernel.

These workloads are inspired by mimicking Vector DB workloads, as this tool has been build concurrently with such exploration, but could be expanded in the future.

The test suite can be run for three different phases: validation, vdb, and stress - see examples below. These phases aim to test the tool under different loads to ensure stability, correctness, and ability to hangle large data files. Be aware that the stress load can take some time and resources to run.

Alongside those there is one suite that needs no root, no device and no workload: tests/placement checks the collector-placement contract (PROFILER_CPUSET) by asserting on make -n output, so it runs anywhere in about a second and is the suite to run on every change. It is what guards backwards compatibility — with PROFILER_CPUSET unset, every layer must emit exactly the command it emitted before the option existed — and it fails if any launch site loses its pinning, which would otherwise leave a collector silently unpinned.

make -C tests/placement validate

Using the tool

For detailed insights into using the tool, please refer to the help pages of the CLI, e.g. by ./run.sh --help. However, we provide some general use examples below. The run.sh script is a simple wrapper, which will use a python script for parsing. This parsed cmd from python will then be passed to the shell.

Be aware that using this tool as part of another tool requires the correct packages. Hence, the easiest strategy is to either propagate the DevBox setup 'up' to the wrapping tool, or simply start FuStIP inside the provided DevBox environment, and then running the relevant workload elsewhere - requiring also stopping FuStIP inside the DevBox.

Note

The CLI of this tool support a --dry option, which will output the parsed command(s) for running the tool. This can be useful for using the tool as part of a pipeline of another tool/setup.

Placing the collectors (PROFILER_CPUSET)

On a machine whose CPUs are all busy, FuStIP's collectors compete with the workload for cores — and losing that competition is silent: the run completes, the CSV looks normal, and events are simply missing. Set PROFILER_CPUSET to a CPU list reserved for the tool and every collector (the three eBPF loaders, their bpftrace equivalents in summary mode, pidstat and the cgroup sampler) is pinned to it:

PROFILER_CPUSET=0,1,48,49 ./run.sh profile start --config ./fustip-config.yml

Important

Reserve whole physical cores — both SMT siblings of each — and exclude them from whatever the workload is pinned to. This matters more than the core count: a collector sharing a physical core with a busy SMT sibling runs at 1.19 M events/s against 2.57 M/s idle (measured, 2×24-core Xeon). Giving it another logical CPU recovers nothing (1.18 M/s); freeing the sibling recovers 2.12×.

Unset or empty means unpinned, which is the right default — on an unloaded machine there is no sibling to lose to, and every existing setup keeps behaving exactly as before. A cpuset the machine cannot honour is refused at startup rather than degrading to an empty capture.

How much to reserve depends on the event rate. As a reference point, three concurrent detailed-mode collectors sustain ~4.83 M events/s on two physical cores and only ~2.37 M/s on one — so one core can be worse than not pinning at all. Compare against your own load: data_quality.drop_pct in each layer's detailed-stats.json reports what was lost.

Every run records its own placement, so "were these counts collected under pinning?" is answered by the result rather than by remembering how it was launched — fustip_overview.txt opens with collectors: pinned to 0,48 or collectors: unpinned, and the test suites print the same in each job banner.

The variable propagates into the test suites too, so the pinned path gets the same end-to-end coverage as the unpinned one — the fio suites still assert the profiler's counts against FIO's, now with the collectors on their reserved cores:

PROFILER_CPUSET=0,48 ./run.sh test validate -m detailed -p fio -d nvme0n1

Using a config file

This tool allows for specifying cli options through a config file also, with the option --config. Below are the formatting of such a yaml file:

mode: detailed

layers:
  - sysstat
  - fs
  - block
  - nvme

comm_filter:
  - fio
  - pidstat

pid_filter:
  - 12345

container_filter:
  - fio-container
  - pidstat-container

dev_filter:
  - nvme0n1
  - nvme1n1

Note, that some options cannot be specified through a config file, and must be passed with other options (see example below).

Start IO Profiling

For all layers for a fio process with nvme dev nvme0n1:

./run.sh profile start -l all -p fio -d nvme0n1

For specified layers:

./run.sh profile start -l block,nvme -p fio -d nvme0n1

For specified commands (without nvme layer):

./run.sh profile start -l block,fs -p fio

For specified containers:

./run.sh profile start -c fio-container

Run detailed mode:

./run.sh profile start -m detailed -p fio -d nvme0n1

Filter by PID (tgid), unioned with comm:

./run.sh profile start -m detailed -P 12345 -d nvme0n1

Clean existing results before running tool:

./run.sh profile start --clean -p fio -d nvme0n1

Using a config file to specify options:

./run.sh profile start --clean --config ./fustip-config.yml

Stop IO Profiling

Important

Stop must be given the same options as the matching start — in particular the same layers (-l) — so every started layer is stopped and its stats generated. In detailed/container mode, stop also writes a consolidated results/fustip_overview.txt. If not specified, this can cause an error as stats (and potentially visualizations) will be generated for layers with no data gathered - as no layers specified defaults to all.

For all layers:

./run.sh profile stop -p fio -d nvme0n1

For specified layers (must match the start):

./run.sh profile stop -l block,nvme -p fio -d nvme0n1

Generate visualizations with results:

./run.sh profile stop --visualize -m detailed -p fio -d nvme0n1

Using a config file to specify options:

./run.sh profile stop --visualize --config ./fustip-config.yml

Run full test suite

Test validation phase with summary mode:

./run.sh test validate -p fio -d nvme0n1

Test all phases with detailed mode and nvme dev nvme0n1:

./run.sh test all -m detailed -p fio -d nvme0n1

Test stress phase with detailed mode against fio container:

./run.sh test stress -c fio-container -d nvme0n1

Test the NVMe layer over io_uring_cmd / NVMe passthrough (--nvme-direct):

FIO_FILE=/dev/ng0n1 ./run.sh test validate --nvme-direct -l nvme -d nvme0n1 -m detailed

--nvme-direct routes the nvme layer to the tests/nvme_direct suite, which runs the same job matrix as block_nvme but with ioengine=io_uring_cmd (cmd_type=nvme). Passthrough submits commands straight to the NVMe driver, bypassing the block layer, so only the nvme layer is profiled. The fio target (FIO_FILE) must be the NVMe generic char device (/dev/ngXnY), while -d/--dev-filter stays the namespace name (nvme0n1) used by the nvme layer's filter and device-info capture.

Warning

Passthrough writes are issued to the whole namespace from LBA 0, ignoring partitions and filesystems — so this suite is destructive and must only be run against a dedicated scratch namespace. As a safeguard, it refuses to start if the target device has mounted partitions or partition-table/filesystem signatures. A stray signature on a genuine scratch device (never a mount) can be overridden with ALLOW_NONEMPTY_DEVICE=1.

Output

Results are written to the directory given by --results-dir (or the RESULTS_DIR env var, which it overrides) — one of the two is required — conventionally results/. It holds one sub-directory per profiled layer plus a top-level overview. The pidstat-based sysstat layer is the same regardless of mode; the eBPF layers (block/nvme/fs) differ:

Mode	Raw output (per eBPF layer)	Aggregated stats
summary	trace.out (bpftrace text)	trace-stats.json (one flat profile)
detailed	detailed.parquet (full per-event trace, ns stamps)	detailed-stats.json (grouped per comm/container)

Per layer:

• sysstat (both modes): cpu.csv / mem.csv / dev.csv (raw pidstat) → sysstat-stats.json — per-process CPU, memory, and disk throughput.
• block / nvme / fs: the stats JSON holds counters (completions, bytes), derived IOPS/throughput (total ÷ active-I/O window), latency/size distributions, inflight + IOPS time-series, and access pattern (sequential vs random). fs keys by syscall (read/write/pread64/pwrite64); block/nvme by request op. In detailed mode nvme nests these per device under each label (per_comm[label].per_disk[<dev>]), so multiple devices are reported separately. When filtered by device, nvme additionally captures device-info.json — a fingerprint of the target device(s) (model, serial, firmware, LBA/sector size, capacity, sector count, MDTS) gathered at capture time via nvme-cli + sysfs.

Detailed/container only:

• data_quality in each stats JSON — ring-buffer drop accounting plus untracked_completions (I/O excluded by the filter, or missed at setup). The dropped events should be 0 (or at least negligibly close), while untracked events are mostly fine - except if they should be included by your specified filter.
• A consolidated results/fustip_overview.txt — a short, human-readable per-layer summary (completions, bytes, IOPS/throughput, access pattern, untracked).