Kernel panics on boot under the CI cross-toolchain (jump to address 0)

[douglasmun]

Summary

Under the nightly CI toolchain (i686-linux-gnu), the kernel does not boot. Every guest boot in the first nightly run panicked with an identical signature, at the point where the scheduler has started all tasks.

This does not reproduce with the local Homebrew i686-elf cross, where the same commit boots and passes harnesses.

Evidence

All four boots that got far enough to start a guest panicked identically:

*** PAGE FAULT ***
Faulting address: 0x00000000
Error code: 0x00000011      (present | instruction fetch)
EIP: 0x00000000
CS: 0x0010
Mode: KERNEL (CPL=0)
...
PT[0]: 0x8000000000000023
  Executable: No (NX set)
*** KERNEL PANIC ***  Kernel mode page fault

EIP=0 with an instruction-fetch error code is a call through a null function pointer. NX on PT[0] is what turns it into a fault rather than executing zeros.

Same signature in all four:

harness	position in run
verify-editor-rowfail.sh	13
verify-exec-frame-leak.sh	18
verify-guard-page-release.sh	22
verify-tcp-rx-counters.sh	60

Not consecutive, and only two of the four use -DTINYOS_FAULT_INJECT, so this is not fault-inject object contamination.

Where it dies

Boot completes all init normally (entropy, ASLR, PAE, IDE, FAT32, all six tasks created). The panic lands just after the scheduler is running everything:

[SUPERVISOR] system task supervisor started [OK]
[EDR DAEMON] Starting EDR background daemon (PID 14583)
[EDR DAEMON] Configuration: scan_interval=500 ticks (5 seconds)
<panic>

One of the four survives a full EDR scan cycle first (Scan complete: 6 processes), so it is shortly after all tasks are live rather than at a fixed instruction.

Why this matters more than the failure count suggests

The first nightly run scored 53 FAIL / 5 INCONCL / 5 PASS. All 5 passes are static source checks that never boot a guest. No CI boot has ever reached the login prompt — grepping every serial log for Enter new root password returns nothing.

So the suite's real state is worse than the fixture failures implied: PR #125 fixes 48 of the 58 non-passing rows (35 × missing disk.img, 13 × dd bs=1m), but the boots that fix unblocks will then hit this.

Ruled out

• Stack: 0x00000000 (ASLR randomized) appears on healthy local boots too — cosmetic reporting, not the cause.
• First task entry is not null — CI reports Task entry: 0x0013539f (local: 0x001381d9). Both plausible.
• Not fault-inject contamination (see above).
• Not flaky — identical signature, every boot, deterministic.

Likely direction

The local and CI crosses differ in ways that have already produced three latent bugs (PR #113: dns.h bool, limits.h #include_next, linker.ld W^X assertion vs default-PIE; and later size_t being unsigned long vs unsigned int). A null function pointer that only materialises under one toolchain suggests a section/relocation difference — e.g. an initialiser array or a weak symbol that the GNU linker places differently — rather than a logic bug.

Suggested first step: build with the CI cross locally (i686-linux-gnu-gcc) and compare objdump/nm output against the i686-elf build, rather than debugging in CI.

Found while classifying the first nightly run's failures for #125.

[douglasmun]

My toolchain hypothesis in the issue body is WRONG. Disproved, not refined.

I reproduced the CI toolchain exactly (Ubuntu 24.04, gcc-i686-linux-gnu, same shim wrappers as the workflow) in Docker and built the same commit:

BUILD CLEAN
kernel.elf: ELF 32-bit LSB executable, Intel 80386, statically linked
716444 bytes

Then compared it against the local Homebrew i686-elf build:

$ cmp /tmp/kernel-ci.elf /tmp/kernel-elf-local.elf
IDENTICAL

The two cross-compilers produce a byte-identical kernel. So there is no section-placement or relocation difference to find, and the suggested objdump/nm diff would have compared a file to itself. Disregard that direction.

The divergence must therefore be in the runtime, not the build. The obvious candidate:

	local	CI
QEMU	11.0.2	8.2.2 (ubuntu:24.04 candidate)
CPU model	Broadwell,+rdrand,+rdseed	same

Same bytes, same -cpu line, different emulator major version. Currently booting the identical ISO under QEMU 8.2 in a container to confirm the panic reproduces there; will post the result either way.

Worth noting for whoever picks this up: if it does reproduce, "works on QEMU 11, panics on QEMU 8.2" does not automatically mean QEMU 8.2 is at fault. A latent kernel bug that the newer QEMU happens to tolerate is at least as likely, and given EIP=0 that is the reading I would weight first.

[douglasmun]

QEMU version is ruled out too — and my whole test setup so far was invalid.

The QEMU 8.2 boot came back clean: same ISO, 90 s, 6 processes alive, 5 EDR scan cycles, KERNEL PANIC count 0. Killed only by my own timeout. So "QEMU 11 tolerates it, 8.2 doesn't" is wrong.

More importantly, I then found the flaw in both of my earlier experiments: my Docker containers were arm64 (Apple Silicon host). I only noticed when grub-pc-bin failed to install with E: Unable to locate package — it is an x86-only package. So:

• the "byte-identical kernel" comparison, and
• the "QEMU 8.2 boots fine" run

were both done in an environment that is not the CI environment. The byte-identical result is probably still meaningful (the cross-compiler targets i686 regardless of host), but I am no longer treating either as settled.

A real difference did show up once I forced --platform linux/amd64: the ISO that grub-mkrescue produces on Ubuntu is 5,838,848 bytes, against 8,941,568 for the local Homebrew i686-elf-grub-mkrescue build. Different bootloader payload, same kernel.

That matters because of what the four panicking harnesses have in common. It is not FAULT_INJECT (only 2 of the 4 use it — my earlier note in this issue was right and a later guess of mine was wrong). It is that all four rebuild the kernel and re-run i686-elf-grub-mkrescue themselves:

harness	rebuild	FAULT_INJECT
verify-guard-page-release	bare make	no
verify-exec-frame-leak	bare make	no
verify-tcp-rx-counters	make clean + FAULT_INJECT	yes
verify-editor-rowfail	make clean + FAULT_INJECT	yes

Every other harness boots the ISO built once by the workflow. These four rebuild it under the shim, and these four are the ones that panic with EIP=0.

One thing I checked and can clear: only i386-pc is present in /usr/lib/grub/ on the runner image, so the no---target invocation does pick i386-pc as the workflow comment claims. That specific worry was unfounded.

Now running the actual reproduction — amd64 container, full CI shim set, bare make, shim grub-mkrescue, QEMU 8.2 — which is the first test I have run that is faithful to CI. Will report the result whichever way it goes.

[douglasmun]

Faithful CI reproduction: does NOT reproduce. Three hypotheses now dead.

Ran the real thing this time — --platform linux/amd64, full CI shim set (i686-elf-gcc wrapper, ld/objcopy/nm/objdump/... symlinks, grub-mkrescue shim), bare make as the harnesses use, then QEMU 8.2 with the harness -cpu Broadwell,+rdrand,+rdseed line:

build ok
mkrescue ok
dist/tinyos.iso  5855232 bytes
=== PANIC COUNT ===
0

120 s, 6 processes, ~10 EDR scan cycles, clean. So the CI toolchain + CI GRUB + CI QEMU together still produce a healthy boot. Ruled out so far: toolchain, QEMU version, GRUB target (only i386-pc is installed on the runner image, so the no---target invocation resolves correctly, as the workflow comment claims).

A correction to something I said earlier in this issue. I suggested the four panicking harnesses might be the FAULT_INJECT ones. They are not — only 2 of 4 use it. What they actually share is that all four rebuild the kernel and re-run i686-elf-grub-mkrescue themselves, while every other harness boots the ISO the workflow built once. That is still the most specific correlation I have, but the reproduction above exercised exactly that path and stayed clean, so it is a correlation without a mechanism yet.

Where the fault actually lands. Three of the four fault immediately after the EDR daemon's config line; the fourth faults after a completed scan cycle:

[EDR DAEMON] Configuration: scan_interval=500 ticks (5 seconds)

*** PAGE FAULT ***
Faulting address: 0x00000000
Error code: 0x00000011   (present | instruction fetch)
EIP: 0x00000000
CS: 0x0010
Mode: KERNEL (CPL=0)
PT[0]: 0x8000000000000023 ... Executable: No (NX set)

Since the fault point differs between runs, it is not a specific instruction in edr_daemon.c — and that file contains no function pointers at all. EIP=0 with an instruction-fetch fault means control was transferred to address 0. The timing (right around when the last daemon starts and the scheduler first rotates a full task set) points at a task being switched into with a corrupted/zero saved EIP, rather than a bad call in daemon code.

The real blocker is that the panic dump cannot localize this. It prints CR3, the faulting address and the page walk, but no ESP, no faulting PID, and no return address from the stack. So there is nothing in the output that identifies which task was entered or where it came from. I would fix that first — adding current PID, ESP, and a couple of stack words to the page-fault handler turns the next nightly into evidence instead of another ambiguous dump. That is a small, self-contained change and useful regardless of this bug.

Not filing a fix for the panic itself yet: I have not reproduced it, and I would rather not guess at a scheduler/context-switch change against a fault I cannot trigger locally.

[douglasmun]

Opened #128: adds the faulting task PID/name, kernel ESP, and an eight-word stack trail to the page-fault dump.

This does not fix the panic — I still cannot reproduce it locally — but it removes the reason the four existing dumps are uninterpretable. Verified by injecting a null call that reproduces the CI signature exactly; the trail resolved back to the injected call site via nm, which is precisely the information the current dumps lack.

Since the fault point varies between runs, the next nightly should distinguish the two candidate mechanisms (bad indirect call vs. context switch into a corrupted saved EIP) rather than leaving both open.

[douglasmun]

Reopening: this was closed automatically when #128 merged, because my commit body phrased its reference in a way GitHub treated as a closing keyword. That was not the intent.

#128 does not fix this panic. It adds the faulting task PID/name, kernel ESP and an eight-word stack trail to the page-fault dump, so the next occurrence is interpretable. The underlying fault is still unreproduced and unexplained: toolchain, QEMU version and GRUB target are all ruled out (see the comments above), and a faithful amd64 reproduction of the CI environment boots clean.

Still open. The next nightly run is the evidence to wait for.

[douglasmun]

Evidence-path fix (no change to the panic itself).

Three harnesses — verify-dns-noprivinsn.sh, verify-dns-rx-counters.sh, verify-netd-arbitration.sh — kept their serial log inside a mktemp -d dir and rm -rf'd it from an EXIT trap. That trap fires on failure too, so if this panic had occurred under one of them, the log carrying #128's new diagnostics (faulting PID/name, kernel ESP, stack words) was deleted before it could be read.

Fixed in #131 via a shared verify/preserve-serial.sh, plus #133 recording the rule. Proven end to end on a real harness: with QEMU stubbed to emit a panic and exit non-zero, the run now preserves

PAGE FAULT at EIP=0x00000000
Task: PID=3 edr_daemon
ESP(kernel): 0x00390018
Stack near ESP: 00000000 c0102f9e
KERNEL PANIC

to $TMPDIR/tinyos-failed-<harness>.serial.log. Previously that file was destroyed. A passing run still leaves nothing behind.

Worth noting for anyone reading back: two of my earlier attempts to force this failure were invalid — the harness rebuilds the ISO itself (so corrupting dist/tinyos.iso was silently repaired), and the wrong-password attempt was meaningless because a first boot sets the root password rather than authenticating. Only the QEMU-stub version actually failed the harness.

Status unchanged: still unreproduced locally, still awaiting a nightly occurrence. The difference is that the next one should now be diagnosable rather than lost.