Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions docs/linux-fault-hosts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Running the Linux-only faults on a Windows laptop (WSL2)

Two fault classes need things Docker Desktop's VM and Windows-side Python don't have:

| Class | Needs | Default |
|---|---|---|
| `disk_full` | `/dev/full` on the host running Python | registered |
| `i2c_sensor_nack` | Linux, root, the `i2c-stub` and `i2c-dev` kernel modules | opt-in: `ORBITLOG_ENABLE_I2C_STUB=1` |

Run these from a WSL2 Ubuntu shell, not from Windows. On Windows, `disk_full` records `trigger_status: failed`
(no `/dev/full`) and `i2c_sensor_nack` is not registered.

## Evidence (17 Oct 2026)

Host fingerprint: `Linux Benweru 6.6.87.2-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Thu Jun 5 18:30:46 UTC 2025 x86_64`
(Windows 11 Pro build 26200, Intel Core Ultra 7 155H, Ubuntu 24.04.3 in WSL2, Docker Desktop 29.8.0). Repo at `a9b93f6`.

**`make dataset N=30 SEED=0` in WSL2** with a running cFS container:

```
cases: 30
by fault_class: cfs_app_crash_restart=5, corrupt_input_file=5, disk_full=4, oom_alloc_failure=4, payload_python_exception=4, sb_pipe_full=4, table_load_validation=4
by trigger_status: triggered=30
by capture: captured=30
```

On Windows the same command left all 5 `disk_full` cases `failed`. In WSL2 the `disk_full` cases contain
`OSError: [Errno 28] No space left on device`.

**`i2c_sensor_nack`, 8 seeds, as root with `ORBITLOG_ENABLE_I2C_STUB=1`:**

```
seed 0: triggered | OSError: [Errno 19] No such device | dmesg lines=0
seed 1: triggered | OSError: [Errno 19] No such device | dmesg lines=1
seed 2: triggered | OSError: [Errno 19] No such device | dmesg lines=0
seed 3: triggered | OSError: [Errno 19] No such device | dmesg lines=0
seed 4: triggered | OSError: [Errno 19] No such device | dmesg lines=0
seed 5: triggered | OSError: [Errno 19] No such device | dmesg lines=0
seed 6: triggered | OSError: [Errno 19] No such device | dmesg lines=1
seed 7: triggered | OSError: [Errno 19] No such device | dmesg lines=0
```

`pytest -m linux_i2c` passed (`1 passed, 85 deselected`).

### How this differs from the Satlyt article

The article shows `tegra-i2c 3180000.i2c: no acknowledge from address 0x36` and `OSError: [Errno 121] Remote I/O error`.
`i2c-stub` is a software SMBus emulator. For an address with no emulated chip it returns **Errno 19 (ENODEV)**, and it
prints no "no acknowledge" kernel warning. The only kernel line it can add is `i2c-stub: Virtual chip at 0x..`.
So this class reproduces the shape of the fault (a sensor read fails, `OSError` in the payload) but **not** the
tegra signature. Dataset consumers and any model evaluation must treat it as an Errno 19 variant.
A real NACK (Errno 121 plus the kernel warning) needs a real controller driver and isn't available on a generic
Linux host.

### Known issue

The `dmesg.txt` window is unreliable: it held 0 or 1 lines across seeds, and the count differed between two runs of the
same seeds (a 3-seed print showed seed 2 with the `Virtual chip` line, the later 8-seed run showed it on seeds 1 and 6).
Don't rely on `dmesg.txt` content until the window capture is fixed.

## Building `i2c-stub` for the WSL2 kernel (no reboot, no admin)

The stock WSL2 kernel has `CONFIG_I2C=y` and `CONFIG_I2C_CHARDEV=m` but no `CONFIG_I2C_STUB`. Build the two modules
against the exact running kernel and load them. The kernel image and `.wslconfig` are not changed.

In the WSL Ubuntu shell, as root (`wsl -d Ubuntu -u root` from Windows needs no password):

```bash
apt-get install -y build-essential flex bison libssl-dev libelf-dev bc dwarves git cpio
git clone --depth 1 --branch linux-msft-wsl-$(uname -r | cut -d- -f1) https://github.com/microsoft/WSL2-Linux-Kernel.git
cd WSL2-Linux-Kernel
zcat /proc/config.gz > .config
scripts/config --module I2C_STUB && make olddefconfig
# LOCALVERSION= (set, empty) stops the build appending "+", so the modules match `uname -r`
make -j"$(nproc)" LOCALVERSION= modules # full build, roughly 20 to 30 minutes on 22 threads
mkdir -p /lib/modules/$(uname -r)/extra
cp drivers/i2c/i2c-stub.ko drivers/i2c/i2c-dev.ko /lib/modules/$(uname -r)/extra/ && depmod -a
modinfo drivers/i2c/i2c-stub.ko | grep vermagic # must start with the output of: uname -r
```

The modules are needed in full because `CONFIG_MODVERSIONS=y` requires the symbol checksums from a complete build.
The `BPF: Invalid name_offset` / `BTF mismatch` messages when loading only affect debug info. Don't load the modules
by hand before running the test: the injector refuses to run if `i2c-stub` is already loaded.

Run the live test:

```bash
sudo ORBITLOG_ENABLE_I2C_STUB=1 .venv/bin/python -m pytest -m linux_i2c -v
```

`sudo` drops the environment variable unless you pass it as above.

### Dataset from WSL2 with Docker Desktop

If Docker Desktop's WSL integration is off for your distro, put a shim on `PATH` that calls the Windows binary:

```sh
#!/bin/sh
exec "/mnt/c/Program Files/Docker/Docker/resources/bin/docker.exe" "$@"
```

The dataset code only uses `docker run/exec/logs/port/inspect`, with no bind mounts, so the shim is enough.
Start the cFS container with the same flags as `make cfs-up` plus `-p 127.0.0.1::1234/udp`, then `make dataset N=30 SEED=0`.
4 changes: 2 additions & 2 deletions docs/tasks/sprint3-linux-fault-injection.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Two more registered fault classes (ids 1 and 2 in `configs/fault_classes.yaml`)
- **Sensor:** "a kernel warning: tegra-i2c 3180000.i2c: no acknowledge from address 0x36 and an OSError: [Errno 121] Remote I/O error triggered during a scheduled camera payload capture". Target hardware is a Jetson Orin Nano.
- **What this machine supports** (probed 16 Oct 2026, Windows 11 + Docker Desktop, VM kernel `6.6.87.2-microsoft-standard-WSL2`):
- cgroup v2 memory caps work. With `--memory=64m --memory-swap=64m`, a 12000×12000×1 allocation got exit 137 with `oom_kill 1` in `memory.events`. Under `ulimit -v 98304` it raised `MemoryError` at `bytearray(n)`.
- The kernel config has `CONFIG_I2C=y` and `CONFIG_I2C_CHARDEV=m`, but **no `CONFIG_I2C_STUB`**, and containers have no `/lib/modules`. **So `i2c_sensor_nack` cannot be triggered here.** It needs a Linux host with the `i2c-stub` module (for example the Pi in plan §9).
- The kernel config has `CONFIG_I2C=y` and `CONFIG_I2C_CHARDEV=m`, but **no `CONFIG_I2C_STUB`**, and containers have no `/lib/modules`. **So `i2c_sensor_nack` cannot be triggered in Docker here.** It needs a Linux host with the `i2c-stub` module. *Update 17 Oct 2026:* it was triggered for real in WSL2 after building the module for the running kernel (8 of 8 seeds, Errno 19, not the article's Errno 121). See [docs/linux-fault-hosts.md](../linux-fault-hosts.md). A Pi isn't needed for this.
- Kernel interfaces: `i2c-stub` takes `chip_addr` (up to 10) and answers SMBus commands only at those addresses (https://docs.kernel.org/i2c/i2c-stub.html). `/dev/i2c-N` needs `i2c-dev`, with `ioctl(I2C_SLAVE, addr)` then SMBus transfers via `I2C_SMBUS` (https://docs.kernel.org/i2c/dev-interface.html).
- No OpenCV dependency: the loader is stdlib Python, so the error text is Python's `MemoryError` or a kernel OOM kill, not OpenCV's `OutOfMemoryError`. That difference is stated, not hidden.

Expand All @@ -35,7 +35,7 @@ Two more registered fault classes (ids 1 and 2 in `configs/fault_classes.yaml`)

## Acceptance criteria
- [ ] Real OOM trigger output for both modes pasted into the PR, with the hardware and kernel fingerprint.
- [ ] `i2c_sensor_nack` either has real trigger output from a Linux host with `i2c-stub`, or the PR states plainly that it hasn't run anywhere yet.
- [x] `i2c_sensor_nack` either has real trigger output from a Linux host with `i2c-stub`, or the PR states plainly that it hasn't run anywhere yet. Real output is in [docs/linux-fault-hosts.md](../linux-fault-hosts.md). Still opt-in, and its signature differs from the article's (Errno 19, no kernel NACK line).
- [ ] `make test` passes. Nothing synthetic is written as log text.

## Out of scope
Expand Down
Loading