Build: Add compiler hardening flags and gate them in CI

[awilczyns]

Summary

Builds the shipped MTL components, DPDK, openh264 and FFmpeg on Linux with hardening flags: stack protector, stack clash protection, CET, _FORTIFY_SOURCE, format checks, full RELRO, a non-executable stack and PIE.

It also fixes a security scan finding on the FFmpeg 7.0 libraries MTL delivers:

• libavcodec, libavdevice, libavformat, libavutil and libswscale all had only partial RELRO;
• three of them had CET shadow stack (SHSTK) without indirect branch tracking (IBT).

A CI check keeps both fixed. 3 commits, 20 files, +481/−13.

Root causes

	Cause	Effect
1	Ubuntu gcc adds -z now only to PIE executables, never to -shared. FFmpeg's configure, DPDK and several MTL meson projects don't pass it themselves.	Their shared objects had partial RELRO, which leaves the GOT writable: 7 of 10 FFmpeg files, 210 of 231 DPDK files, all 9 GStreamer plugins, st22_avcodec and 2 sample plugins.
2	FFmpeg 7.0 x86inc.asm marks every NASM object SHSTK only and emits no endbr64. ld ANDs the CET property over all input objects.	libavcodec, libavutil and libswscale (also libavfilter and libswresample) lose IBT, however the C code is compiled. No compiler or linker flag can fix this, and upstream has no fix (checked up to 9.0.2 and today's master).
3	Nothing in the tree sets the compile flags. The build only looks hardened on Ubuntu, because Ubuntu gcc enables most of them by default. RHEL-family gcc enables none.	On Rocky/RHEL (for example docker/rocky9.dockerfile), MTL, DPDK and FFmpeg got no stack protector, CET, FORTIFY or PIE, and only libmtl.so had full RELRO.

What each commit does

Build: Add compiler hardening flags

• Meson projects. The meson projects of the shipped MTL components get the same block: lib, app, plugins, plugins/st22_avcodec, manager, tests, tests/tools/RxTxApp, tests/tools/gstreamer_tools, ecosystem/gstreamer_plugin, ecosystem/obs_mtl/linux-mtl and gpu_direct.
  • The block adds -fstack-protector-strong -fstack-clash-protection -fcf-protection=full through get_supported_arguments, which drops a flag a compiler doesn't support instead of failing the build.
  • It adds -Wformat -Wformat-security -Werror=format-security without a probe. meson probes each flag on its own, and on RHEL gcc the probe of -Werror=format-security fails with '-Wformat-security' ignored without '-Wformat', which would drop it. Every gcc and clang MTL builds with accepts the three.
  • It links with -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack.
  • Each subdirectory is its own meson project, so there is no shared place for the block.
  • Projects that also build for Windows guard the block with if not is_windows.
  • The existing -z now -z relro in lib/meson.build is replaced by the block, not duplicated.
• _FORTIFY_SOURCE. -D_FORTIFY_SOURCE=2 is added only when optimizing and when neither the compiler nor c_args/cpp_args sets a level.
  • Ubuntu 24.04 gcc sets 3. gcc 13 would take =2 without a warning and lower the level, so it must not be added there.
  • The level is read from the compiler's predefined macros (cc -O2 -dM -E). meson 1.11 and later run cc.get_define() with -U_FORTIFY_SOURCE, so it can't see the level.
• USDT. systemtap's dtrace -G compiles the USDT provider object with CFLAGS from the environment only. Without -fcf-protection=full there, the object has no .note.gnu.property, and ld drops IBT and SHSTK from libmtl.so. That is what happened on RHEL, where gcc doesn't enable CET by default.
• PIE. b_pie=true in app, manager, tests and tests/tools/RxTxApp.
• DPDK. script/build_dpdk.sh passes the flags with -Dc_args, -Dc_link_args and -Db_pie=true. -Dc_args replaces CFLAGS, so the script prepends the caller's CFLAGS/LDFLAGS.
• FFmpeg and openh264. ecosystem/ffmpeg_plugin/build.sh passes the flags:
  • to FFmpeg (all versions) with --extra-cflags, --extra-ldflags and --extra-ldexeflags=-pie;
  • to openh264 through CFLAGS/LDFLAGS in the environment, which its Makefile appends to.
• Docs. doc/build.md §4.5 documents the flags and the manual DPDK command.

Fix: Add CET IBT to the FFmpeg 7.0 x86 assembly

New ecosystem/ffmpeg_plugin/7.0/0002-x86-add-Intel-CET-IBT-support.patch (4 FFmpeg files, +38/−4). The existing patch loop applies it after 0001. It makes the assembly IBT-correct first, and only then marks it IBT:

File	Change
libavutil/x86/x86inc.asm	endbr64 at every cglobal/cvisible entry, and the property note becomes IBT and SHSTK. Both apply only with NASM ≥ 2.15.01; older NASM builds exactly as before.
libavcodec/x86/vvc/vvc_mc.asm	The jump table points at new .wN_ibt: labels, each an endbr64 that falls through into .wN. That keeps the endbr out of the loop.
libavcodec/x86/mlpdsp_init.c	The two computed jumps into the unrolled filter become notrack jmp, which is what GCC emits for switch tables.
libswscale/x86/hscale_fast_bilinear_simd.c	The MMXEXT scaler code generated at runtime starts with endbr64 (x86-64 only). The size pass counts those 4 bytes.

endbr64 is a NOP on CPUs without CET, and .text grows by 0.8 %. OpenBSD, which enforces IBT in user space, has similar patches in its FFmpeg port, including the same .wN_ibt labels in vvc_mc. dav1d MR 1775 does the same for x86inc.

Setting only the note bit, or linking with -z ibt, would produce a false mark that faults under IBT enforcement.

Ci: Check compiler hardening of the dependency caches

.github/scripts/ci/check-hardening.sh PATH... checks every x86-64 executable and shared object under the given paths. It requires all of:

• GNU_RELRO and BIND_NOW (full RELRO);
• a non-executable GNU_STACK, with any alignment;
• PIE (ET_DYN);
• x86 feature: IBT, SHSTK.

It is fail-closed. Every check needs a positive match, and every given path must hold at least one x86-64 ELF file, so a missing or empty path fails, also next to a good one.

• validate-cache.sh runs it on the dpdk, mtl, ffmpeg, gstreamer and plugins caches. So their builds and restores are checked, without a new workflow.
• In branch mode (the branch input of build.yml and the pytest workflows), the trees come from another ref, HEAD, and CI from the workflow commit. validate-cache.sh checks them with HEAD's own check-hardening.sh, and not at all when HEAD predates it, so a ref is held to its own hardening. A git error fails the validation.
• The rocky9 image build runs it on /install, RxTxApp, MtlManager and KahawaiTest, so a regression that only RHEL-family gcc shows fails Docker Build. .dockerignore lets the script into the build context, and a change to the script reruns the image builds.
• libopenh264 is exempt from the CET check only. Its assembly has no CET mark and no flag can add one. RELRO, BIND_NOW, NX and PIE are still required of it. A process that loads it runs without CET, as on main.

Verification

Toolchain: Ubuntu 24.04, gcc 13.3, nasm 2.16.01 and binutils 2.42 (the CI runners' toolchain). All builds ran in a private clone.

The CI flow on main and on this branch (build-dependencies.sh, then validate-cache.sh):

Cache	main	This branch
dpdk	210 of 231 fail (no BIND_NOW)	0 of 231
mtl	2 of 6 fail	0 of 6
ffmpeg	7 of 10 fail	0 of 10, and 0 of 11 on a runner that builds openh264
gstreamer	9 of 9 fail	0 of 9
plugins	1 of 1 fail	0 of 1

On main, the ffmpeg check finds what the scan found, and two libraries the scan doesn't list:

• libavcodec, libavutil and libswscale report no BIND_NOW IBT/SHSTK;
• libavformat and libavdevice report no BIND_NOW;
• libavfilter and libswresample, not in the scan, report no BIND_NOW IBT/SHSTK.

Rocky 9 (gcc 11.5). The rocky9 image build passes the check: libmtl.so has x86 feature: IBT, SHSTK, and 54 of 54 compile commands carry -Werror=format-security. The same dockerfile on the tree without the format and USDT changes fails it with libmtl.so is not hardened: no IBT/SHSTK.

The check really fails:

• Removing a single property from a hardened binary makes the check fail and name exactly that property. Tested with -z norelro, -z lazy, -z execstack, -no-pie, -fcf-protection=none, and with an SHSTK-only NASM object linked in.
• A missing, empty or symlinked path fails, alone and next to a path that passes.
• The openh264 exemption matches by filename only: the same file renamed fails with no IBT/SHSTK, and main's openh264 still fails with no BIND_NOW.

Branch mode:

• Before the fix, build.yml dispatched on this branch with branch set to today's main restored main's caches and failed Evaluate cache results: 210 of 231 dpdk files no BIND_NOW (run 37025547441).
• With the fix, in worktrees laid out as overlay-tests leaves them and forged trees that pass every other check:
  • an unhardened tree passes when HEAD is main, and fails with no BIND_NOW when HEAD is this branch or a later ref;
  • a hardened tree passes with hardening: 1 ELF files checked, also when the workflow commit carries a stricter check-hardening.sh that no tree passes;
  • evaluate-caches.sh gives the same results;
  • outside a git work tree, dpdk fails with git's error while jpegxs still validates.
• Each of four broken variants fails these tests: the probe inside the if (a git error skips the check), an always-skipped check, a probe of the index instead of HEAD, and the workflow commit's check-hardening.sh instead of HEAD's.

The IBT is real, not just a mark:

• Static audit of every NASM object:

  • no patch: 2527 functions without endbr64;
  • x86inc hunk only: 0 functions, but 28 jump-table targets in vvc_mc without it;
  • full patch: 0 and 0.

• Intel SDE with IBT enforcement (sde64 -spr -cet 1 -cet_raise 0). ENDBRANCH errors with the x86inc hunk only, then with the full patch:

  Workload	x86inc hunk only	Full patch
  checkasm, all 9729 tests	294	0
  MLP decode test program	384160	0
  MLP decode through ffmpeg	160	0
  swscale fast-bilinear test program	2880	0

  Without any patch, a full run gives 65845.

• The output is the same with and without the patch: checkasm passes, and the MLP and swscale checksums match.

The flags are really applied (compile flags leave no mark in the ELF):

• 100 % of compile_commands.json entries carry the flags.
• The binaries import __stack_chk_fail and __*_chk.
• The _FORTIFY_SOURCE block, in all 11 projects built with --werror, with each meson version the CI uses with that compiler (0.61.2 to 1.12.1):
  • gcc 11.4 (Ubuntu 22.04) and gcc 13.3 (Ubuntu 24.04) keep their own level 2 and 3, and get no -D_FORTIFY_SOURCE=2;
  • gcc 11.5 (Rocky 9) gets level 2;
  • -D_FORTIFY_SOURCE=1 in CFLAGS stays 1;
  • a debug build gets none.

Nothing broken:

• Unit tests: 690 of 690 pass.
• Acceptance smoke, e810 leg: 12 passed, 2 skipped (static skip marks), 0 failed.
• The MinGW build with b_pie links, and the result runs under wine.

Notes for reviewers

• Cache rebuild. The first CI run rebuilds every dependency cache except ice: script/build_dpdk.sh is in the dpdk cache key, at the root of the chain, and ecosystem/ffmpeg_plugin/build.sh is in the jpegxs key.
• New error. -Werror=format-security is new for DPDK, openh264 and FFmpeg. All three build clean with it.
• Cost. -z now moves symbol binding to load time. On Ubuntu the generated C code doesn't change, because those compile flags are already gcc defaults there. On RHEL it does change, and that cost isn't benchmarked.
• Severity. Linux enforces only the shadow stack in user space today, not IBT. The partial RELRO was the exploitable part. The IBT change makes the mark correct and ready for enforcement.

Not in this PR

• FFmpeg CVEs. The same scan lists 20 CVEs against FFmpeg 7.0.3:

  • release/7.0 already contains the fixes for 12 of them;
  • 4 don't reach the libraries MTL builds (never in 7.0's code, the zmqsend tool, or the TensorFlow filter);
  • 4 are real: CVE-2026-40962, CVE-2026-30997, CVE-2025-22921 and CVE-2025-10256.

  FFmpeg 8.1.3 is the lowest release that has all the fixes and isn't matched by NVD's version ranges. Moving to it means porting the MTL avdevice plugin and patch 0002 to FFmpeg 8 (libavcodec.so.62), so it is a separate PR.

• openh264 CET. It needs endbr64 in openh264's assembly, and upstream openh264 has none.

• FFmpeg 6.1 and 4.4. Their assembly has no CET note, so they build without IBT and SHSTK.

• tools/set_tai_offset, and libbpf and libxdp from script/build_ebpf_xdp.sh. They keep their own build flags and aren't checked. set_tai_offset isn't installed; the images ship libbpf and libxdp.

• SVT-JPEG-XS, the ICE kernel module, Windows. Their build flags are unchanged. The Windows equivalents (/guard:cf, /CETCOMPAT) aren't covered.

• Unpinned FFmpeg source. build.sh still downloads the moving release/7.0 branch head, not a tag.