fchmodat2 AT_EMPTY_PATH: no such file or directory while creating device nodes on kernel 5.10

[gerwinjansen]

Description

I'm trying to upgrade k3s on my QNAP. Version v1.32.1-k3s1 runs fine, but v1.33.13-k3s2 fails. Together with Claude.ia I'm debugging the problem and it results in this bug report. The text of this report is crafted by Claude, and carefully reviewed by me, although I do not understand all the details. Hope it gives enough information for you to make up your mind.

fchmodat2(AT_EMPTY_PATH) fails during device-node creation on a Linux 5.10 kernel, and (unlike the /proc-based fallback path that's supposed to handle kernels without AT_EMPTY_PATH support) the failure happens on the initial fchmodat2 call itself, before the fallback logic is even reached. This has been reproduced identically on runc v1.4.2 and runc v1.5.1 (the current latest release), on the same host, changing only the runc binary.

This looks related to #5038, which hit a similarly-shaped error (fchmodat ... no such file or directory) on SUSE 15 SP4 (kernel 5.14, also pre-6.6).

One difference from #5038: their fallback path (pathrs.ProcThreadSelfOpen → fchmodat on the resulting /proc/self/fd/N) was reached and then failed with ENOENT. In my case, the failure message indicates the first fchmodat2(AT_EMPTY_PATH) call itself returns something other than EINVAL/EOPNOTSUPP (the only two errnos that trigger the fallback), so the fallback branch may never be entered at all. Both cases may share a root cause, or may be two distinct gaps in the same kernel-compatibility logic.

Both my case and the SUSE case involve running inside a containerized/nested environment (I'm running k3s-in-Docker on a QNAP NAS; the SUSE report doesn't specify but the maintainer's failure to reproduce on a plain VM is suggestive). It may be worth considering whether the trigger requires more than just "kernel < 6.6" in isolation — possibly something about /proc visibility or namespace configuration in these nested setups.

Steps to reproduce the issue

1. Start a privileged container from that same image to isolate the test from Kubernetes entirely:

   docker run -it --privileged --entrypoint /bin/sh rancher/k3s:v1.33.13-k3s2

2. Inside the container, start containerd manually and pull a test image:

   /bin/containerd &
   sleep 2
   ctr -a /run/containerd/containerd.sock images pull docker.io/library/busybox:latest

3. Attempt to run a container using the native snapshotter (required to avoid an unrelated nested-overlayfs limitation in this Docker-in-Docker setup):

   ctr -a /run/containerd/containerd.sock run --rm --snapshotter=native docker.io/library/busybox:latest test /bin/true

4. To retest with a different runc build, replace /bin/runc inside the container with the desired release binary and repeat 3)

Describe the results you received and expected

Received (identical on both runc v1.4.2 and v1.5.1):

ctr: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: configuring container /dev: error creating device nodes: update new c device inode /dev/null file mode: fchmodat2 AT_EMPTY_PATH: no such file or directory

Expected: The container starts successfully.
If I repeat the steps to reproduce with rancher/k3s:v1.32.1-k3s1, it all works fine. That image ships:

~ # containerd --version
containerd github.com/k3s-io/containerd v1.7.23-k3s2
~ # runc --version
runc version 1.2.4
commit: 688fdd8
spec: 1.2.0
go: go1.23.4
libseccomp: 2.5.5

Again, if I replace just the binary of runc with version 1.5.1 it fails as above.

Since AT_EMPTY_PATH support for fchmodat2 was only added in Linux 6.6, and this kernel is 5.10, I'd expect the existing pre-6.6 fallback path (added in commit 01de9d6, part of the CVE-2025-52565 fix, present since runc 1.2.8/1.3.3/1.4.0-rc.3) to handle this transparently — as it apparently does on other pre-6.6 kernels, since this is presumably a common configuration that hasn't produced widespread reports.

What version of runc are you using?

Inside the rancher/k3s:v1.33.13-k3s2 container

runc version 1.4.2
commit: v1.4.2-0-gc241c0b
spec: 1.3.0
go: go1.25.11
libseccomp: 2.6.0

Also tested with:

runc version 1.5.1
commit: v1.5.1-0-g8f2685a47
spec: 1.3.0
go: go1.25.12
libseccomp: 2.6.0
libpathrs: 0.2.5

The container ships:

containerd github.com/k3s-io/containerd/v2 v2.2.5-k3s1.33

Cgroup: v1

Host OS information

QNAP TS-251D running

NAME="QTS"
VERSION="5.2.10 (20260731)"
ID=qts
PRETTY_NAME="QTS 5.2.10 (20260731)"
VERSION_ID="5.2.10"

Docker version 27.1.2-qnap8, build b8cbe19`

Host kernel information

Linux NAS 5.10.60-qnap #1 SMP Fri Jul 31 04:04:00 CST 2026 x86_64 GNU/Linux

[cyphar]

As with #5038, the most helpful information would be to use retsnoop to find exactly where the error is coming from.

ENOENT from fchmodat2(AT_EMPTY_PATH) seems unlikely to have been caused by the same underlying issue as #5038 -- if AT_EMPTY_PATH was not set then you would expect ENOENT when passing an empty path string but we do set it, and AT_EMPTY_PATH does not depend on procfs so all of that speculation is not really useful.

[gerwinjansen]

Given this:

# ls -la /sys/kernel/btf/
/bin/ls: cannot access /sys/kernel/btf/: No such file or directory
# ls -la /sys/kernel/debug/
total 0
dr-xr-xr-x  2 admin administrators 0 2026-08-20 21:38 ./
drwxr-xr-x 14 admin administrators 0 2026-08-12 15:16 ../
# zcat /proc/config.gz
zcat: /proc/config.gz: No such file or directory

Claude told me that retsnoop and ftrace/kprobes wont'work.

I used strace version 7.2 like this:

strace -f -yy -e trace=openat,fchmodat2,close,mknod,mknodat -o /tmp/containerd-strace2.log /bin/containerd &
sleep 2
ctr -a /run/containerd/containerd.sock run --rm --snapshotter=native docker.io/library/busybox:latest test /bin/true
grep -n -E 'openat|fchmodat2|close|mknod' /tmp/containerd-strace2.log | grep -B5 'fchmodat2'
1268:319   close(13</1/task/1>)              = 0
1269:319   close(16</1/task/1/fd/15>)        = 0
1270:319   close(11</run/containerd/io.containerd.runtime.v2.task/k8s.io/test/rootfs>) = 0
1271:319   mknodat(15</run/containerd/io.containerd.runtime.v2.task/k8s.io/test/rootfs/dev>, "null", S_IFCHR|0666, makedev(0x1, 0x3)) = 0
1272:319   openat(15</run/containerd/io.containerd.runtime.v2.task/k8s.io/test/rootfs/dev>, "null", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 11</run/containerd/io.containerd.runtime.v2.task/k8s.io/test/rootfs/dev/null>
1273:319   fchmodat2(11</run/containerd/io.containerd.runtime.v2.task/k8s.io/test/rootfs/dev/null>, "", 020666, AT_EMPTY_PATH) = -1 ENOENT (No such file or directory)

The Claude generated this program to reproduce the issue:

#define _GNU_SOURCE
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <errno.h>

#ifndef __NR_fchmodat2
#define __NR_fchmodat2 452
#endif
#ifndef AT_EMPTY_PATH
#define AT_EMPTY_PATH 0x1000
#endif

int main() {
    int fd = open("/dev/null", O_PATH | O_NOFOLLOW | O_CLOEXEC);
    if (fd < 0) { perror("open"); return 1; }

    long ret = syscall(__NR_fchmodat2, fd, "", 0666, AT_EMPTY_PATH);
    printf("fchmodat2 returned %ld, errno=%d (%s)\n", ret, errno, strerror(errno));
    return 0;
}

When I run this:

# /tmp/repro
fchmodat2 returned -1, errno=2 (No such file or directory)

Is this helpfull?
I will go out for a holiday, so my next comment is just over a week from now.