Recurring IBM Block CSI NodeStageVolume failures: lun not valid with mixed-LUN multipath map on node

[fontexD]

Summary

We are seeing recurring IBM Block CSI mount failures (2-15/day over 29 days) that do not self-heal via kubelet retries.

For the active incident below, controller attach succeeded, but NodeStageVolume repeatedly failed on LUN validation.

Environment

• Kubernetes version: Client Version: v1.29.2; Server Version: v1.35.1
• IBM Block CSI operator image: quay.io/ibmcsiblock/ibm-block-csi-operator:1.13.2@sha256:44d76626c656d0ca4bb84fa8ee51217868b6713787e46e5715505d290011547f
• IBM Block CSI node image: quay.io/ibmcsiblock/ibm-block-csi-driver-node:1.13.2@sha256:290a430df5dfd7a5cae33339d2b87a38a3d92216ff01f24cc03c098214a8abce
• IBM Block CSI controller image: quay.io/ibmcsiblock/ibm-block-csi-driver-controller:1.13.2@sha256:ef1ee31e402038bebf517b9020016d10ff882e4bb0dbccb7a3c94390c527e4aa
• Sidecars in use:
• csi-addons-replicator: quay.io/ibmcsiblock/csi-block-volumereplication-operator:v0.9.3@sha256:b2b7dd017ab540042ece5b0288e4ffbfe75132a14d9b2319957fe1993795edcc
• csi-attacher: registry.k8s.io/sig-storage/csi-attacher:v4.9.0@sha256:5aaefc24f315b182233c8b6146077f8c32e274d864cb03c632206e78bd0302da
• csi-provisioner: registry.k8s.io/sig-storage/csi-provisioner:v4.0.1@sha256:bf5a235b67d8aea00f5b8ec24d384a2480e1017d5458d8a63b361e9eeb1608a9
• csi-resizer: registry.k8s.io/sig-storage/csi-resizer:v1.14.0@sha256:5e7cbb63fd497fa913caa21fee1a69f727c220c6fa83c5f8bb0995e2ad73a474
• csi-snapshotter: registry.k8s.io/sig-storage/csi-snapshotter:v8.3.0@sha256:bc7be893ecc3ad524194aa6573b2f5c06cd469bdf21a500ab6c99c2ba1c4d64d
• csi-volume-group: quay.io/ibmcsiblock/csi-volume-group-operator:v0.9.3@sha256:085347d46774a422af008d502924c2cd16c94473e83832c6da6896625f149e46
• livenessprobe: registry.k8s.io/sig-storage/livenessprobe:v2.16.0@sha256:88092d100909918ae0a768956cf78c88bc59cd7232720f7cdbdfb5d2e235001e
• csi-node-driver-registrar: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.14.0@sha256:5244abbe87e01b35adeb8bb13882a74785df0c0619f8325c9e950395c3f72a97
• livenessprobe: registry.k8s.io/sig-storage/livenessprobe:v2.16.0@sha256:88092d100909918ae0a768956cf78c88bc59cd7232720f7cdbdfb5d2e235001e
• Storage backend: IBM 2145 (FlashSystem), iSCSI + multipath + xfs
• Affected node: k8s-worker-5
• Healthy comparison node: k8s-worker-6

Affected workload and volume

• Namespace: ai
• Pod: bizdev-6bc467c444-nqzfw
• PVC: chatbot-indexes-pvc
• PV: pvc-c4c1087e-8837-4862-b2d1-d84c598c4e7c
• VolumeHandle: SVC:198;60050763808104EA28000000000000E4
• VolumeAttachment: csi-17c48ab79ec93f03b3278380a94b916af9b09b8fde90ed51cced35e58a7d0e5e
• PublishContext LUN: 113

Symptom details

Pod events:

• SuccessfulAttachVolume
• repeated FailedMount with MountVolume.MountDevice failures

CSI node log error:

• 2026-07-13 12:53:24,71312 ERROR [27329] [-] (driver.go:89) - GRPC error: rpc error: code = Internal desc = lun not valid, storage lun: 113, linkedPath: /sys/devices/platform/host34/session2/target34:0:0/34:0:0:18 to device: sdac

Key evidence

On affected node k8s-worker-5, multipath maps include mixed LUN members on at least one dm map.

Observed maps summary:

Node         maps  mixed_maps  failed_faulty_lines
k8s-worker-5 15    3          20
k8s-worker-6 17    0          2

Affected node mixed maps:
- MIXED 360050763808104ea280000000000003a dm-3 -> 29,74
- MIXED 360050763808104ea28000000000000cc dm-13 -> 136,126
- MIXED 360050763808104ea28000000000000e4 dm-15 -> 118,113,121,18

Expected behavior

After attach and host rescan, NodeStageVolume should converge and mount using only published LUN paths for the volume.

Actual behavior

NodeStageVolume retries continue to fail because validation inspects dm slave paths that still include stale mismatched LUN members. Retries do not clean stale path members, so issue persists until manual intervention or rescheduling.

Request for IBM

1. Confirm whether this is a known issue in 1.13.2.
2. Provide official safe remediation for stale mixed-LUN path membership on active nodes.
3. Provide fixed version or backport guidance.
4. Clarify expected driver behavior when stale failed paths with mismatched LUN remain in a discovered dm map.

node-driver-logs-window.txt