fsck OOM-crashloop on csi-local-node after AKS node upgrade (10 TiB ext4, 600 Mi limit)

[jmclong]

What happened

After an AKS node-image upgrade, the csi-local-node DaemonSet's driver container is OOM-killed in a tight crashloop while staging a previously formatted local volume. The pod consuming the PVC stays in ContainerCreating indefinitely with MountVolume.MountDevice failed ... context deadline exceeded.

The volume is ~10 TiB of ext4 holding many small files (high inode count). Each pod restart re-runs fsck from the top, hits the 600 Mi limit again, and gets killed. We could not break the loop without intervention.

Mitigation we used: delete and recreate the PVC (data was cache and could be rebuilt). We don't believe this should be the only option.

Why it happens

internal/csi/mounter/mounter.go embeds k8s.io/mount-utils.SafeFormatAndMount, whose formatAndMountSensitive flow calls checkAndRepairFilesystem -> fsck -a synchronously before mount on any already-formatted rw volume (upstream code). For ext4, fsck -a dispatches to e2fsck -p (preen mode). On a clean unmount this is effectively a no-op, but after an ungraceful detach during the node upgrade the filesystem state was dirty and e2fsck entered repair mode.

Peak e2fsck memory scales with the working set of inode bitmaps, the icount structures (pass 1), and dx_dir / orphan-list state (pass 2). For a 10 TiB ext4 with the default bytes-per-inode = 16384 (~640M inodes, mke2fs.conf defaults) holding millions of small files, repair-mode RSS routinely exceeds the 600 Mi container limit set in charts/latest/values.yaml:

driver:
  limits:
    memory: 600Mi
  requests:
    cpu: 10m
    memory: 60Mi

The fact that e2fsprogs ships an e2fsck.conf(5) scratch_files option specifically to spill these structures to disk on memory-constrained systems is itself evidence that this working set can grow well past sub-gigabyte limits on large filesystems. Azure/AKS#4682 has an independent report of the same OOM on a 12,500 GiB azuredisk volume against the same 600 Mi node-DaemonSet limit.

Two compounding factors turn the OOM into an unbreakable crashloop:

1. No timeout on fsck. SafeFormatAndMount.checkAndRepairFilesystem calls mounter.Exec.Command("fsck", "-a", source).CombinedOutput() with no context.Context, no timer, and no semaphore - unlike the format path right next to it which does honour mounter.formatTimeout. Combined with the OOM kill, every restart re-enters fsck from scratch and the loop never terminates.
2. No skip-fsck escape hatch at the mount-utils layer or as a CSI driver flag / StorageClass option. Once you're in this state, the only way out is recreating the volume or running e2fsck out-of-band on the host.

Prior art

This pattern has been reported against multiple CSI drivers and is not yet fixed upstream:

• [BUG] azuredisk container in csi-azuredisk-node is OOM killed during running fsck AKS#4682 - azuredisk-csi-node OOM-killed during fsck (12,500 GiB volume, 600 Mi limit; closed-as-stale in June 2025 without a code fix).
• [BUG] csi-azuredisk-node is OOMing causing all pods with PV to Pending forever AKS#4421 - earlier related thread on memory pressure in the same node DaemonSet.

Repro

1. Provision a local.csi.acstor.io PVC, ~10 TiB, ext4.
2. Fill it with many small files (push inode count into the hundreds of millions).
3. Trigger an ungraceful node detach (AKS node-image upgrade, or reboot -f on the node).
4. After the node comes back, observe the csi-local-node driver container OOMKilled with the last log line being Checking for issues with fsck on disk: /dev/....
5. Pod consuming the PVC stays in ContainerCreating; kubectl get events shows MountVolume.MountDevice failed ... DeadlineExceeded.

Expected behavior

Any one of:

• The node DaemonSet has enough memory headroom to complete e2fsck -p on realistic large volumes; or
• There is a documented, supported way to skip or bound the in-line fsck (per-StorageClass parameter, per-PV annotation, or driver flag); or
• The OOM does not produce an unbreakable crashloop - e.g. fsck timeout + the driver fails the NodeStageVolume RPC cleanly with a clear error message so the operator can intervene.

Environment

• Kubernetes version: 1.35.0
• local-csi-driver version: v0.2.10 (we have not retested on v0.2.14 though nothing has changed there)
• Volume size / fs type / approx file count: ~10 TiB / ext4 / millions of small files

[NedAnd1]

According to https://issuetracker.google.com/issues/338125229, mkfs & fsck should be able to make progress after being OOMKilled, but if we're not observing that even after a really long period of time, that could be related to the behavior change in k8s w/ nodes using cgroups v2 where a process like fsck or mkfs running out of memory causes the entire container to OOMKilled unless singleProcessOOMKill is enabled (https://communities.sas.com/t5/SAS-Communities-Library/cgroups-v2-changes-in-Kubernetes-1-28-and-why-it-impacts-your/ta-p/944364).
In my mind the MemoryQoS feature (https://kubernetes.io/blog/2023/05/05/qos-memory-resources/) could help us out.
We could also test out what happens when we explicitly set memory.oom.group to 0 within the csi node container.