Skip to content

feat(image): guard against kernel/modules version mismatch - #673

Open
HarryWaschkeit wants to merge 1 commit into
omnect:mainfrom
HarryWaschkeit:feat/verify-kernel-module-consistency
Open

feat(image): guard against kernel/modules version mismatch#673
HarryWaschkeit wants to merge 1 commit into
omnect:mainfrom
HarryWaschkeit:feat/verify-kernel-module-consistency

Conversation

@HarryWaschkeit

Copy link
Copy Markdown
Contributor

Problem

BitBake's per-task sstate/setscene mechanism decides cache reuse independently, per task, based only on whether a task's own declared taskhash matches an already-cached object. For linux-yocto style kernel recipes this means do_package_write_ipk/do_populate_sysroot can be sstate-restored from an older cached object while do_compile/do_sign/do_deploy execute fresh in the same build — each decision is individually 'correct' per BitBake's own bookkeeping.

This is compounded by the kernel-yocto SCC/KMETA merge tree embedding a non-reproducible git describe hash into LOCALVERSION/vermagic at build time, which BitBake's signatures never track. The result: a kernel Image and its shipped kernel-modules package can silently originate from two different kernel builds, producing an image whose modules fail to load at runtime (module version magic mismatch) — with no error anywhere in the build.

Observed directly in omnect-os-build/tauril2,gateway-devel build 107 (2026-07-10): confirmed via the Concourse build log that do_package_write_ipk was a setscene cache-restore from build 104, while do_kernel_metadata/do_compile/do_sign/do_compile_kernelmodules/do_deploy all executed fresh — and via device evidence (uname -r vs. the /lib/modules directory name showing different non-reproducible hash suffixes).

See #670 for the part of this specific incident's root cause that's fixable at the BitBake-signature level (a spurious DISTRO_FEATURES vardep on do_sign/do_deploy).

Fix

This PR does not attempt to fix the caching behavior itself — that isn't reliably possible within BitBake's pre-execution signature model when the actual divergence stems from unTracked non-determinism in an upstream task's real output. Instead, it adds an IMAGE_POSTPROCESS_COMMAND check (classes/omnect-verify-kernel-modules.bbclass, inherited by omnect-os-image.bb) that:

  1. Extracts the kernel version banner string (Linux version ...) from the deployed kernel/fitImage blob in DEPLOY_DIR_IMAGE.
  2. Compares it against the /lib/modules/<version> directory name actually present in the built rootfs.
  3. Fails the build (bb.fatal) if they don't match.

This is independent of and complements #670: it catches this class of mismatch regardless of cause, turning a silent runtime failure into a loud, immediate CI build failure — so an inconsistent image like build 107's never reaches a test device or the field again.

Limitations / follow-ups

  • Works for the uncompressed arm64 Image format used by our current machines (verified against phygate-tauri-l-imx8mm-2's KERNEL_IMAGETYPE = "Image"). A compressed KERNEL_IMAGETYPE (zImage/uImage with gzip/lz4) would need the check extended to decompress first — it currently warns and skips rather than false-failing in that case.
  • Does not identify which task was the stale one, only that a mismatch exists — for full incident triage the sstate siginfo comparison approach used in the original investigation is still the recommended method.
  • Genuinely orthogonal to fix(kernel): narrow do_sign/do_deploy DISTRO_FEATURES vardep to secureboot #670: can be merged independently, before, after, or without it.

Adds an image-postprocess check that compares the kernel version
string embedded in the deployed kernel Image/fitImage against the
version directory name shipped in /lib/modules, and fails the build
if they don't match.

This is a build-time consistency guard, not a caching fix: BitBake's
per-task sstate/setscene mechanism can independently sstate-restore
do_package_write_ipk/do_populate_sysroot from an older cached object
while do_compile/do_sign/do_deploy execute fresh in the same build.
Combined with the kernel-yocto SCC/KMETA merge tree's non-reproducible
LOCALVERSION suffix (untracked by BitBake signatures), this can produce
an internally consistent-looking (per BitBake bookkeeping) but actually
broken image where the kernel and its modules originate from different
kernel builds and modules silently fail to load at runtime.

Observed directly in omnect-os-build/tauril2,gateway-devel build 107
(2026-07-10). See PR omnect#670 for the part of the root cause (a spurious
DISTRO_FEATURES vardep on do_sign/do_deploy) that could be fixed at the
signature level. This check is independent of and complements that fix:
it catches this class of mismatch regardless of cause, turning a silent
runtime failure into a loud CI build failure.

Current implementation extracts the kernel version by scanning the
deployed kernel/fitImage blob for the "Linux version ..." banner
string, which works for the uncompressed arm64 Image format used by
our machines; compressed KERNEL_IMAGETYPEs would need the check
extended to decompress first (it currently warns and skips rather than
false-failing in that case).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant