TLDR: Currently the happy path for volume deletion, where an NVMe volume is deleted successfully by the node that owns it, produces a large volume of error logs. The volume of logs scales with the number of nodes in the pool.
Ideally, if the owning node still exists, non-owning nodes should recognise that they have no work to do and that nothing is in an unexpected state - silently.
What happens
On a standard Azure Container Storage v2 install for local NVMe, non-replicated volumes always use the hyperconverged webhook (this is the default and, for non-replicated volumes, is not configurable). With the webhook active, PVs carry no node affinity, so all DaemonSet driver instances process every DeleteVolume request. If I understand docs/design/pv-cleanup.md correctly, this is by design as it enables the overall system to recover from failed nodes. Great!
docs/design/pv-cleanup.md says a non-owning instance should check whether the node is deleted and return success if so. But in the normal case, where the owning node still exists and successfully deletes the volume itself, the other instances currently fall through to FailedPrecondition: Volume is on a different node. Each logs this as an ERROR, writes a VolumeFailedDelete Warning event onto the PV object, logs an Unhandled Error, and retries with backoff.
The volume is deleted correctly. This is just a log/event noise issue, but it scales with cluster size and occurs on the default install path - so it seems worth raising and addressing!
Evidence (log lines involving a single PVC, 65-node pool, 107s window)
| Metric |
Value |
| Total log lines |
1,567 |
| ERROR-level lines |
768 |
| Owning node lines (correct/functional lifecycle) |
31 |
| Other 64 nodes |
1,536 |
VolumeFailedDelete events on the PV |
256 |
Each non-owning node emits ~24 lines (failures=0..3) in ~1.4s. Per-volume line count is uniform across ~600 volumes (~127–328 each): fixed per-node cost × node count.
Representative non-owning node:
[ERROR] "Volume deletion failed" err="... FailedPrecondition ... Volume is on a different node <owner>"
[ERROR] "Event occurred" ... type="Warning" reason="VolumeFailedDelete"
[ERROR] "Unhandled Error" err="error syncing volume ... FailedPrecondition ..."
"Retrying syncing volume" failures=0..3
Environment
- Azure Container Storage v2 (
microsoft.azurecontainerstoragev2 cluster extension — the documented Terraform/CLI install), local NVMe storage class
local-csi-driver:v0.2.17, csi-provisioner v5.2.0, --node-deployment --strict-topology=true- AKS, 65-node pool, generic ephemeral (non-replicated) volumes → hyperconverged webhook active by default
- Confirmed on PVs:
.spec.nodeAffinity unset, .spec.csi.volumeAttributes."localdisk.csi.acstor.io/selected-initial-node" populated
- Workload: short-lived CI build agents → high volume churn
Suggested fix
Extend the non-owning-node handling described in pv-cleanup.md so an instance that doesn't own the volume returns cleanly in both cases: node deleted (already designed) and node present / already handled by the owner — without logging FailedPrecondition at ERROR or emitting a VolumeFailedDelete event. Since this is the default local-NVMe path, it affects any cluster following the standard install.
TLDR: Currently the happy path for volume deletion, where an NVMe volume is deleted successfully by the node that owns it, produces a large volume of error logs. The volume of logs scales with the number of nodes in the pool.
Ideally, if the owning node still exists, non-owning nodes should recognise that they have no work to do and that nothing is in an unexpected state - silently.
What happens
On a standard Azure Container Storage v2 install for local NVMe, non-replicated volumes always use the hyperconverged webhook (this is the default and, for non-replicated volumes, is not configurable). With the webhook active, PVs carry no node affinity, so all DaemonSet driver instances process every
DeleteVolumerequest. If I understanddocs/design/pv-cleanup.mdcorrectly, this is by design as it enables the overall system to recover from failed nodes. Great!docs/design/pv-cleanup.md says a non-owning instance should check whether the node is deleted and return success if so. But in the normal case, where the owning node still exists and successfully deletes the volume itself, the other instances currently fall through to
FailedPrecondition: Volume is on a different node. Each logs this as an ERROR, writes aVolumeFailedDeleteWarning event onto the PV object, logs anUnhandled Error, and retries with backoff.The volume is deleted correctly. This is just a log/event noise issue, but it scales with cluster size and occurs on the default install path - so it seems worth raising and addressing!
Evidence (log lines involving a single PVC, 65-node pool, 107s window)
VolumeFailedDeleteevents on the PVEach non-owning node emits ~24 lines (
failures=0..3) in ~1.4s. Per-volume line count is uniform across ~600 volumes (~127–328 each): fixed per-node cost × node count.Representative non-owning node:
Environment
microsoft.azurecontainerstoragev2cluster extension — the documented Terraform/CLI install), local NVMe storage classlocal-csi-driver:v0.2.17, csi-provisionerv5.2.0,--node-deployment --strict-topology=true.spec.nodeAffinityunset,.spec.csi.volumeAttributes."localdisk.csi.acstor.io/selected-initial-node"populatedSuggested fix
Extend the non-owning-node handling described in
pv-cleanup.mdso an instance that doesn't own the volume returns cleanly in both cases: node deleted (already designed) and node present / already handled by the owner — without loggingFailedPreconditionat ERROR or emitting aVolumeFailedDeleteevent. Since this is the default local-NVMe path, it affects any cluster following the standard install.