What happened
local-csi-driver (localdisk.csi.acstor.io) is documented as cloud-agnostic ("compatible with various environments, including Azure and AWS"), but on an AWS EKS GPU node it discovers zero NVMe disks and publishes no CSIStorageCapacity, even though a raw EC2 instance-store NVMe disk is present and visible inside the driver container. Any PVC on the driver's StorageClass stays Pending forever.
Environment
|
|
| Driver image |
mcr.microsoft.com/acstor/local-csi-driver:v0.2.18 (gitCommit 050d007526e34ee6eb7117687336f9012887e20d, buildId 20260623.1) |
| Chart |
oci://localcsidriver.azurecr.io/acstor/charts:0.2.18 |
| Platform |
AWS EKS 1.33, Amazon Linux 2023 (6.12.92-122.166.amzn2023.x86_64), containerd 2.2.4 |
| Instance type |
g5.4xlarge (us-west-2), 1× 600 GB EC2 NVMe instance store |
| Consumer |
KAITO workspace controller (bundles this chart) |
Root cause
The node driver filters candidate disks by matching a hardcoded Azure device model string, Microsoft NVMe Direct Disk. AWS instance-store NVMe reports a different model, so it is filtered out and totalNVMeDisks=0.
Node driver container log (csi-local-node-* / driver):
startup.go:85 "no available disks found" logger="startup-diagnostic" totalNVMeDisks=0 nonLVM2FormattedDisks=0
Event Warning NoDiskAvailable: "No NVMe disks matching the expected model
(Microsoft NVMe Direct Disk) were found on this node. This can happen when the
node pool uses a VM SKU with ephemeral OS disk enabled, which consumes the NVMe
disk for the OS. Consider using a VM SKU with additional NVMe disks, or disable
ephemeral OS disk on the node pool."
The disk is clearly present and visible inside the driver container:
$ lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL
NAME SIZE TYPE FSTYPE MOUNTPOINT MODEL
nvme0n1 100G disk Amazon Elastic Block Store # EBS root
|-nvme0n1p1 100G part xfs /var/lib/kubelet
...
nvme1n1 558.8G disk Amazon EC2 NVMe Instance Storage # <-- raw, unformatted, unmounted
$ cat /sys/block/nvme1n1/device/model
Amazon EC2 NVMe Instance Storage
$ cat /sys/block/nvme1n1/size
1171875000 # 1171875000 * 512B = 600 GB
So the discovery predicate — not the absence of a disk — is the problem. The Azure-specific model string makes the driver effectively Azure-only, contradicting the documented AWS support.
Impact / observable symptoms
$ kubectl get csidrivers
NAME ... STORAGECAPACITY ...
localdisk.csi.acstor.io ... true ...
$ kubectl get csistoragecapacity -A
(no resources) # <-- driver advertises capacity but publishes none
# => any PVC on the driver's StorageClass never binds; consuming pod stuck Pending
# ("waiting for a volume to be created / no capacity").
Expected behavior
On AWS nodes with an EC2 instance-store NVMe disk (model Amazon EC2 NVMe Instance Storage, raw/unformatted), the driver should discover the disk, prepare it (LVM/format as it does on Azure), and publish a non-zero CSIStorageCapacity so PVCs bind.
Reproduction
- EKS 1.33, AL2023, node
g5.4xlarge (has instance-store NVMe).
- Install
local-csi-driver chart 0.2.18.
- Create a PVC on its StorageClass (
provisioner: localdisk.csi.acstor.io, volumeBindingMode: WaitForFirstConsumer) and a pod consuming it scheduled onto the GPU node.
- Observe: PVC never binds;
kubectl get csistoragecapacity -A empty; node driver logs show totalNVMeDisks=0 + NoDiskAvailable.
What happened
local-csi-driver(localdisk.csi.acstor.io) is documented as cloud-agnostic ("compatible with various environments, including Azure and AWS"), but on an AWS EKS GPU node it discovers zero NVMe disks and publishes noCSIStorageCapacity, even though a raw EC2 instance-store NVMe disk is present and visible inside the driver container. Any PVC on the driver's StorageClass staysPendingforever.Environment
mcr.microsoft.com/acstor/local-csi-driver:v0.2.18(gitCommit050d007526e34ee6eb7117687336f9012887e20d, buildId20260623.1)oci://localcsidriver.azurecr.io/acstor/charts:0.2.186.12.92-122.166.amzn2023.x86_64), containerd 2.2.4g5.4xlarge(us-west-2), 1× 600 GB EC2 NVMe instance storeRoot cause
The node driver filters candidate disks by matching a hardcoded Azure device model string,
Microsoft NVMe Direct Disk. AWS instance-store NVMe reports a different model, so it is filtered out andtotalNVMeDisks=0.Node driver container log (
csi-local-node-*/driver):The disk is clearly present and visible inside the driver container:
So the discovery predicate — not the absence of a disk — is the problem. The Azure-specific model string makes the driver effectively Azure-only, contradicting the documented AWS support.
Impact / observable symptoms
Expected behavior
On AWS nodes with an EC2 instance-store NVMe disk (model
Amazon EC2 NVMe Instance Storage, raw/unformatted), the driver should discover the disk, prepare it (LVM/format as it does on Azure), and publish a non-zeroCSIStorageCapacityso PVCs bind.Reproduction
g5.4xlarge(has instance-store NVMe).local-csi-driverchart0.2.18.provisioner: localdisk.csi.acstor.io,volumeBindingMode: WaitForFirstConsumer) and a pod consuming it scheduled onto the GPU node.kubectl get csistoragecapacity -Aempty; node driver logs showtotalNVMeDisks=0+NoDiskAvailable.