feat(node): native NVMe/FC ANA multipath support (discovery, stage/publish/expand/stats, path accounting) - #877
Draft
binhnguyenduc wants to merge 8 commits into
Conversation
NVMe/FC NodeStageVolume relied on the kernel AEN to enumerate a newly mapped namespace; RescanDevices was a no-op for NVMe. When the AEN is missed, the namespace never appears, multipathd shows no map, and staging fails with "Couldn't find multipath device for volumeID". Implement RescanDevices to run `nvme ns-rescan` on the array's connected controllers, matched by traddr against the array target ports (0x-normalized so publish-context ports match list-subsys output). Idempotent and non-fatal: a rescan that finds nothing, a list-subsys failure, or a transient per-controller failure does not fail the stage, since GetMpathDevice retries discovery afterwards. NVMe-only; SCSI/iSCSI rescan paths are unchanged.
…Executer-backed helper Two identical copies read /sys/module/nvme_core/parameters/multipath directly via os.ReadFile (device_connectivity.isNvmeCoreMultipathEnabled and NodeUtils.IsNativeNVMeMultipathEnabled), so neither was fake-able in unit tests. Export a single IsNvmeCoreMultipathEnabled that reads through the Executer interface; both call sites delegate to it. Adds a table-driven test for the detector and an IoutilReadFile expectation to the GetMpathDevice tests that now route the read through the mock Executer. No behavior change.
…ipath Under native NVMe multipath (nvme_core.multipath=Y) the kernel presents an NVMe/FC volume as a single namespace-head /dev/nvmeXnY with no dm device, so the multipathd-based discovery returns nothing and NodeStageVolume fails with "Couldn't find multipath device". Add a native-aware GetMpathDevice on the NVMe/FC connectivity: when native mode is detected, resolve the head by the volume's NGUID via the stable /dev/disk/by-id/nvme-eui.<nguid> symlink, with a sysfs /sys/block/nvme*/wwid fallback (normalizing dashes/prefix/case) and a bounded retry while udev settles after the namespace rescan. Under dm-multipath it delegates to the existing shared discovery unchanged. Scoped to NVMe/FC by dispatch, so SCSI and iSCSI volumes on a native-NVMe host are unaffected.
NodeStageVolume unconditionally called GetSysDevicesFromMpath, which reads /sys/block/<dev>/slaves — a device-mapper construct absent for a native NVMe namespace head, so mount staging errored on native-multipath hosts. Branch on DevicesAreNvme: for native NVMe skip both physical-path discovery and lun validation (the kernel manages ANA paths and ValidateLun is already a no-op for NVMe); dm-multipath (SCSI/iSCSI/non-native NVMe) keeps validating the paths behind the dm device. Existing stage tests gain the NotNVMe expectation and a new case asserts the native branch skips both calls.
…tem) NodeGetVolumeStats resolved devices only via dm-multipath, so on native NVMe multipath both paths failed: GetBlockVolumeStats used the dm-only shared GetMpathDevice, and IsVolumePathMatchesVolumeId queried multipathd for a device with no dm map. Promote the native NGUID resolver to an exported DiscoverNativeNamespaceDevice and reuse it for block stats; add a native branch to IsVolumePathMatchesVolumeId that matches the mounted namespace head's sysfs wwid against the volume NGUID instead of multipathd. dm-multipath unchanged.
The native NodeExpandVolume branch skipped the dm-multipath resize (correct) but did nothing to refresh the namespace size, so ExpandFilesystem read the stale pre-expand size and silently under-grew unless a kernel AEN had already landed. Add RescanNvmeNamespaceForResize: read the namespace head's subsysnqn and run nvme ns-rescan on the controllers of that subsystem (best-effort, non-fatal), wired via NodeUtils.RescanNvmeNamespace before the filesystem grow. dm and non-native NVMe expand paths unchanged.
…h native-aware resolver
NodeStageVolume resolved the device via the connectivity-dispatched (native-aware)
GetMpathDevice, but NodeUnstageVolume, NodePublishVolume (raw block),
NodeExpandVolume, and NodeGetVolumeStats called the shared dm-only helper
directly — so on a native-multipath node they took the multipathd path, waited
for a dm device that never appears, and failed ("Couldn't find multipath device").
NodeExpandVolume was the one a canary hit; the others were latent (unstage passed
only via its idempotent not-found swallow).
Add NodeUtils.DiscoverMpathDevice: in native mode resolve the namespace head by
NGUID (single-shot, the device already exists post-stage), falling back to dm
discovery on a miss so a SCSI/iSCSI volume on a native-NVMe host still resolves
(the native and dm device namespaces are disjoint — no false match). Route all
four non-stage call sites through it. DiscoverNativeNamespaceDevice gains a
maxRetries arg (stage keeps the full settle budget; post-stage uses 1) and no
longer sleeps after the final attempt.
…ccounting The kernel's nvme list-subsys emits FC addresses with a "0x" hex prefix, while the publish-context array targets and sysfs host ports are unprefixed. The live- path map was keyed with the prefixed form and looked up with the unprefixed form, so countLivePathsForSubsystem always returned 0 even with 8 live ANA paths. Staging still worked (EnsureLogin is advisory and host autoconnect brings paths up), but the driver logged a false "0 live paths" error on every stage and ran a redundant discover/connect storm each time, and a genuine fabric outage would be indistinguishable. Normalize both sides (strip 0x, lowercase) via the existing normalizeTraddr when building and comparing keys. Also treat nvme connect's "already connected" output as success, since that is the steady state under host autoconnect. Adds white-box tests for normalizeTraddr and countLivePathsForSubsystem.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds full native-ANA multipath support for NVMe-oF/FC to the node driver. On arrays that present ANA (native
nvme_coremultipath,multipath=Y), device discovery for stage/publish/expand/stats must target the native NVMe namespace head (/dev/nvmeXnY, resolved by NGUID) instead of a dm-multipath device. This series makes every node-side discovery path native-aware while preserving the existing dm-multipath (non-native) behaviour unchanged.Based directly on
release-1.14.0(no divergence) and leaves the driver version unchanged.What changed
nvme_coremultipath) vs non-native (dm), used across the lifecycle.NodeExpandVolumeso the device sees the grown capacity (avoids under-growth).0xtraddr prefix inEnsureLoginso publish-context target ports matchnvme list-subsysoutput regardless of the kernel hex prefix (prevents mis-counting live paths).Tests
Unit coverage lands with the series:
device_connectivity_nvmeofc_native_test.go(native discovery/lifecycle), additions todevice_connectivity_nvmeofc_internal_test.goanddevice_connectivity_nvmeofc_test.go, plusnode_test.go/node_utils_test.gofor native routing in the node service.make testandgo test -race ./node/...both pass on Linux against the module's pinned deps (go 1.25.10,k8s.io/mount-utils v0.35.0); no data races.Notes
common/config.yamland the Dockerfiles are untouched.nvme_coremultipath is active.Draft: opening for maintainer review of the approach before finalizing.