Skip to content

local-csi-driver v0.2.18 fails to discover AWS EC2 instance-store NVMe (hardcoded "Microsoft NVMe Direct Disk" model match) #529

Description

@rambohe-ch

What happened

local-csi-driver (localdisk.csi.acstor.io) is documented as cloud-agnostic ("compatible with various environments, including Azure and AWS"), but on an AWS EKS GPU node it discovers zero NVMe disks and publishes no CSIStorageCapacity, even though a raw EC2 instance-store NVMe disk is present and visible inside the driver container. Any PVC on the driver's StorageClass stays Pending forever.

Environment

Driver image mcr.microsoft.com/acstor/local-csi-driver:v0.2.18 (gitCommit 050d007526e34ee6eb7117687336f9012887e20d, buildId 20260623.1)
Chart oci://localcsidriver.azurecr.io/acstor/charts:0.2.18
Platform AWS EKS 1.33, Amazon Linux 2023 (6.12.92-122.166.amzn2023.x86_64), containerd 2.2.4
Instance type g5.4xlarge (us-west-2), 1× 600 GB EC2 NVMe instance store
Consumer KAITO workspace controller (bundles this chart)

Root cause

The node driver filters candidate disks by matching a hardcoded Azure device model string, Microsoft NVMe Direct Disk. AWS instance-store NVMe reports a different model, so it is filtered out and totalNVMeDisks=0.

Node driver container log (csi-local-node-* / driver):

startup.go:85 "no available disks found" logger="startup-diagnostic" totalNVMeDisks=0 nonLVM2FormattedDisks=0
Event Warning NoDiskAvailable: "No NVMe disks matching the expected model
(Microsoft NVMe Direct Disk) were found on this node. This can happen when the
node pool uses a VM SKU with ephemeral OS disk enabled, which consumes the NVMe
disk for the OS. Consider using a VM SKU with additional NVMe disks, or disable
ephemeral OS disk on the node pool."

The disk is clearly present and visible inside the driver container:

$ lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL
NAME             SIZE  TYPE FSTYPE MOUNTPOINT       MODEL
nvme0n1          100G  disk                         Amazon Elastic Block Store   # EBS root
|-nvme0n1p1      100G  part xfs    /var/lib/kubelet
...
nvme1n1        558.8G  disk                         Amazon EC2 NVMe Instance Storage   # <-- raw, unformatted, unmounted

$ cat /sys/block/nvme1n1/device/model
Amazon EC2 NVMe Instance Storage
$ cat /sys/block/nvme1n1/size
1171875000            # 1171875000 * 512B = 600 GB

So the discovery predicate — not the absence of a disk — is the problem. The Azure-specific model string makes the driver effectively Azure-only, contradicting the documented AWS support.

Impact / observable symptoms

$ kubectl get csidrivers
NAME                      ...  STORAGECAPACITY  ...
localdisk.csi.acstor.io   ...  true             ...

$ kubectl get csistoragecapacity -A
(no resources)            # <-- driver advertises capacity but publishes none

# => any PVC on the driver's StorageClass never binds; consuming pod stuck Pending
#    ("waiting for a volume to be created / no capacity").

Expected behavior

On AWS nodes with an EC2 instance-store NVMe disk (model Amazon EC2 NVMe Instance Storage, raw/unformatted), the driver should discover the disk, prepare it (LVM/format as it does on Azure), and publish a non-zero CSIStorageCapacity so PVCs bind.

Reproduction

  1. EKS 1.33, AL2023, node g5.4xlarge (has instance-store NVMe).
  2. Install local-csi-driver chart 0.2.18.
  3. Create a PVC on its StorageClass (provisioner: localdisk.csi.acstor.io, volumeBindingMode: WaitForFirstConsumer) and a pod consuming it scheduled onto the GPU node.
  4. Observe: PVC never binds; kubectl get csistoragecapacity -A empty; node driver logs show totalNVMeDisks=0 + NoDiskAvailable.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions