SPEC-73 tiering: bdev_tier composite + relocate primitives + nexus heat + raid5f hardening#13
Open
rducom wants to merge 42 commits into
Open
SPEC-73 tiering: bdev_tier composite + relocate primitives + nexus heat + raid5f hardening#13rducom wants to merge 42 commits into
rducom wants to merge 42 commits into
Conversation
- vbdev_tier.h: band-table data model (band=disk, tier class, state), composite layout (mirrored md region [0,md_end) + concat data bands), per-band failure isolation, lifecycle + relocate-quiesce API. - Makefile (overlay template). - Dockerfile: COPY + register tier in DIRS-y / BLOCKDEV_MODULES_LIST / DEPDIRS. WIP: vbdev_tier.c (io path/translate/copy/md-mirror/superblock) + RPC next.
- vbdev_tier.c: concat translate, mirrored md region (RAID1 across 2 bands), per-band failure isolation (-EIO on degraded band, never global is_degraded), channels (per-band base_ch), lifecycle create/add_band/retire/register/delete, relocate-quiesce co-design hook for M2b. - vbdev_tier_rpc.c: bdev_tier_create/add_band/register/retire_band/get_bands/delete.
…acro fix (SPEC-73 M1) - vbdev_tier.h: on-disk tier_superblock + tier_sb_band structs (à la bdev_raid_sb), TIER_SB_* constants, per-band phys_offset, sb_blocks/seq/registered fields, fix SPDK_BDEV_MAX_NAME_LENGTH -> TIER_BDEV_NAME_LEN (CI build error). - vbdev_tier.c: reserve sb_blocks at start of each base bdev; geometry computes phys_offset for md-mirror bands (A/B) vs plain concat; io path offsets every physical access by phys_offset (data) / sb_blocks (md region). WIP next: async superblock write/read + examine-based self-assembly (wwn validation).
- vbdev_tier_sb.c: serialize + CRC32c, async write-all-bands (seq bump), async read+validate from a base desc (building block for examine assembly). - persist superblock to every band on register + retire_band. - Makefile: add vbdev_tier_sb.c. WIP next: examine-based self-assembly (read sb on boot, wwn-validate, auto-create).
Physical LBA of a cluster on the bs_dev (bdev_tier composite), so the control-plane can map clusters to bands/tiers. New public blob API (patch 0004), conflict-free with 0001-0003 (lib/blob only).
Single lib/blob patch (conflict-free with 0001-0003): - spdk_blob_get_cluster_lba (M2a): cluster -> physical LBA. - spdk_blob_claim_cluster_in_range (M2b): claim free cluster in a band's LBA range. - spdk_blob_relocate_commit (M2b): md-thread swap-in-place + persist extent + release old after durability (invariants A/B; native replay reclaims orphans).
Per allocated cluster -> physical LBA (via spdk_blob_get_cluster_lba, patch 0004). Control-plane maps LBA->band/tier from its band table. Stacked on 0002 (lvol).
- vbdev_tier_relocate_copy: direct base-bdev copy between bands (bypasses composite/quiesce). - blob patch 0004: + spdk_blob_release_cluster_lba (abort path). - lvol patch 0005: bdev_lvol_relocate_cluster RPC = claim -> quiesce(old) -> copy(base-direct) -> commit(md-thread swap) -> unquiesce, with abort cleanup. - Dockerfile: DEPDIRS-bdev_lvol += bdev_tier ; lvol CFLAGS -I tier.
…EC-73 M3) raid5f_chunk_read_complete: on !success, recompute geometry and reconstruct the chunk from survivors+parity instead of failing. Enables degraded reads during a bdev_tier band failure. (NULL-peer double-fault guard = follow-up.)
Patch 0007 (stacked on 0001): hot-path hook in raid_bdev_submit_request + dense per-region read/write counters (lossy, INV-52) + RPCs vbdev_nexus_enable_heat/get_heat/disable_heat. Read async by the brain; never on the I/O path (INV-T2).
…t free-at-destruct - vbdev_tier add_band: reject a wwn already present in the composite (duplicate disk → double-counted capacity / corrupt concat geometry). - raid5f reconstruct read: bail out -EIO if a SECOND member is unavailable (NULL base channel) — raid5 tolerates one fault; avoids submitting to a missing base bdev. - raid_bdev_free: raid_tier_heat_destroy() at nexus teardown (reclaim the heat sketch).
rducom
force-pushed
the
feat/spec-73-tiering
branch
from
June 21, 2026 14:43
ae265c3 to
d85cce3
Compare
Live turing deploy surfaced this: on a 1-disk node the md region had no second mirror band, so tier_route_rw returned -EIO for the md range → GPT/raid examine failed → lvolstore init failed (could not initialize blobstore). Now the md region is RAID1-mirrored only when a second band exists; a single-band composite serves md single-copy on its one band (no redundancy is possible with one disk anyway). The geometry already places md at [sb_blocks, sb_blocks+md) on band A.
Data-safety guard (G-M2b-3): bdev_lvol_relocate_cluster now rejects a cluster of a snapshot blob (read-through-shared by clones via back_bs_dev). Moving it without clone back-dev re-resolution (§7.3, out of v1) would corrupt the clones' view.
After the destination write, read it back and CRC32c-compare against the source (snapshotted under quiesce, so stable). A silent media/write corruption is caught at relocate time with -EIO rather than surfacing later through the upper-layer redundancy. Disk-to-disk copy ⇒ no accel memory-copy offload applies; the integrity check does.
…y guard) Relocating a snapshot blob's cluster is now permitted (§7.3). Read-through coherence for clones holds: the copy+swap run under quiesce of the composite OLD range (a clone's read-through resolves to the snapshot's bs_dev = this composite at old_lba, held until the atomic swap, then resumes onto new_lba), and relocate_commit runs md-thread-serialized with snapshot create/delete/decouple.⚠️ UNVALIDATED until the S-D-snap spike runs on multi-tier hardware.
…eback range repair Repairs lost ranges of a degraded member without raid internals: per stripe-aligned range, read from the raid bdev (M3 reconstructs the degraded chunk from survivors+parity for raid5f; raid1 reads a healthy mirror), then write the buffer back (recomputes parity / rewrites mirrors), repairing the relocated/replacement band. Public bdev read/write API only; QD=1 background; INV-T2 (data stays in SPDK). Works for both replica and EC.⚠️ UNVALIDATED until exercised on EC + a real disk failure.
…band, swap detection) vbdev_tier_assemble_band places a band at EXPLICIT stored geometry (band_id/lba_start/num_blocks/state/ is_md) so the CSI agent reassembles a composite identically across reboots and can detect a physically swapped disk (live wwn != slot's stored wwn). bdev_tier_read_sb returns the on-disk superblock (the full band table incl. wwn per slot). Completes A2 beyond the wwn-sort (which already fixes reorder).
… guard, F4 per-blob serialize, F6 remap-nocopy, F8 flush-before-verify, F10 atomic SB seq
…nc, SB durability - C3: hot-remove drains per-reactor channels then closes the desc (SPDK REMOVE contract), persists DEGRADED; new bdev_tier_resync_md RPC rebuilds a replacement md leg under an md-range quiesce with channel propagation before activation. - M1/T-2: md fan-out never completes the orig_io while a submitted leg is in flight. M2/T-6: async read failover to the mirror leg. M3: a failed md write leg is degraded+persisted; write succeeds off the surviving leg. - M4/W9: -ENOMEM completes NOMEM everywhere (broken queue_io path removed). - M5: SB generation reserved at write_all entry, fan-outs serialized and coalesced, DEGRADED bands excluded (unfreezes the seq); F-6 FLUSH per copy; F-4 LE-only guard; F-1 on-disk ABI locked by static asserts. - M6/T-3/W7: assemble_band validates band_id/state bounds, overlap, capacity, duplicate wwn. W1: double-register guard. T-4/T-7/MJ6: retire_band is async (ack after persistence), drains before close, refuses md-mirror bands. m1/m2/m6/m7 minors + header comments fixed. - rpc: md_num_blocks>0 and cluster_blocks<=u32 validation.
- CBT-1/CBT-2: epoch_freeze/epoch_close refuse -EBUSY while a rebuild is RUNNING on the epoch (freeing bitmap_frozen under the scanner was an UAF). - c5: higher-generation epoch_open cannot rip a REBUILDING epoch. - CBT-4: the rebuild FLUSHes the target before COMPLETED (a lacunary member believed synced after a power cut was silent corruption). - New: anti-double-rebuild guard in the shared engine (covers the legacy bdev_cbt_partial_rebuild path the CSI actually calls); rebuild_id stamped before the first I/O (sync completion left get_rebuild_status -ENOENT); find_active_for_epoch discriminates by cbt node; dead override_ranges decoder removed; c4: no silent recreation with a virgin bitmap on base-bdev reappearance. - D3: dead healthy-clear poller and backends_healthy machinery removed; bitmap clearing is reset-driven (documented).
…ENOSPC, S-2 socket — patches 0001-0010 - 0001: CBT-6 channel-promotion failure propagates + full unwind (never ONLINE with a NULL channel); P-5 runtime slot check; membership rollback. - 0003: registry entry preallocated before pause dispatch (no orphan pause on OOM); UUID nonce (P-6); loud TTL clamp. - 0004: expose spdk_blob_freeze_io/unfreeze_io (C1 foundation); m8 assert. - 0005: C1 — relocate runs under a BLOB freeze (composite quiesce replayed held writes to the pre-swap LBA = lost write); N-2 blob pinned by an own open-ref; P-2 lvol-on-composite check; N-7/W6 remap requires DEGRADED source + ACTIVE destination; m9 placement scan budget. - 0007: M7 release/acquire publish of the heat sketch (arm64), per-region TSC recency (no shared tick cacheline), N-8 grace-period free on disable. - 0008: C2 — every repair chunk runs under a channel-owned LBA range lock (exported from lib/bdev; host writes held, replayed after write-back); P-3 stripe alignment validated, heap ranges; N-6 REMOVE aborts. - NEW 0009: thin-pool ENOSPC surfaces as NVMe CAPACITY_EXCEEDED end-to-end; raid1 keeps the member and fails the write to the consumer (C-1). - NEW 0010: JSON-RPC unix socket chmod 0600 after bind (S-2). Patches regenerated from a clean per-patch commit series (format-patch --zero-commit); all 10 apply cleanly on vanilla v26.01.
- module/bdev/tier/test: host-compiled unit tests of the PRODUCTION code (#include vbdev_tier_sb.c) under ASAN+UBSAN — on-disk ABI, serialize/ validate roundtrip, corruption/version/byte-swapped-magic rejection, geometry inlines, seq semantics. Mock SPDK headers, no SPDK build needed. - workflow: tests run as a lint-job step on every PR (types now include 'opened'); compute-tag/build depend on it — a red test blocks the image. - Dockerfile: upstream tag pinned to commit 2ef883e and verified at build (a moved tag fails the build). Verified end-to-end in podman: 10/10 patches apply, zero-warning build, spdk_tgt links.
…, patch tooling, RPC contract
On-disk superblock v2 (clean break — v1 is NOT readable, no migration: the
format never shipped publicly, every redeploy is wipe+reinstall):
- magic TIERSB02, header 256 B, band 192 B; cluster_blocks widened to u64 (F-3).
- F-5: the 256 KiB reserve is two 128 KiB A/B slots; generation seq N is written
to slot N%2 and flushed alone, so a torn write damages at most one slot;
readers take the highest-seq valid slot (tier_sb_select).
- F-2: generation_uuid (composite-instance fencing, minted at create, exposed by
read_sb), created_epoch_sec, 104 B header + 32 B/band reserve. Static asserts
192/256/12544. docs/FORMAT-tier-superblock.md.
evariops_get_capabilities (PR2): boot_id (CSI detects a target restart →
volatile state lost), tier_sb_version, method list, single_reactor_assumed (D5).
bdev_lvol_relocate_clusters (PF3): batch relocate under ONE amortized blob
freeze over N clusters (≤4096); partial success is a 200 with {relocated,
requested, error}. verify flag (PF4) forwards to each copy — skips the C5
read-back on trusted media. Regenerated patch 0005.
Tooling: scripts/patches.sh (check/apply/regen/verify, reads the pinned SHA from
the Dockerfile) + patches/README.md (order, sed wiring, upstreaming) — closes
the manual-hunk-editing risk (U-3/U-5/U-6). docs/RPC-CONTRACT.md (PR1): per-RPC
preconditions/idempotence/crash behavior + invariants P1-P5, D5, CBT-7.
Host tests extended (A/B slot selection, torn-slot fallback, binary golden
header, u64 cluster_blocks). Validated end-to-end in podman: 10/10 patches
apply, zero-warning build, spdk_tgt links.
… patches In-repo module/script fixes: - tier: fix leg_io double-free on md-read failover retry failure - tier: add_band TIER_MAX_BANDS bound + registered guard (base_ch OOB) - tier: fail channel-create on NULL base io_channel - tier: re-validate dst_band->desc across the relocate copy chain (hot-remove) - tier: io_type_supported consults base bands for UNMAP/FLUSH/WRITE_ZEROES - tier: overflow-safe range/capacity checks in assemble_band - tier: multi-band FLUSH/UNMAP/WRITE_ZEROES fan-out (fixes whole-device flush) and RESET propagation to all base bands - tier(rpc): drop stale u32 cluster_blocks cap (v2 SB is u64); drop dead capacity_blocks/used_blocks fields - cbt: clear out_rebuild_id on start_rebuild error; c5 takeover guard keys on an actually-running rebuild (fixes permanent -EBUSY after a completed rebuild) - patches.sh: check is non-destructive (scratch index), NUL-safe patch list Out-of-tree patch fixes (regenerated via patches.sh regen): - 0003 nvmf-pause: re-resolve subsystem by NQN before TTL resume (TTL UAF) - 0004 blob-relocate: roll back in-memory L2P swap on sync failure (corruption) - 0007 raid-heat: guard blocklen==0 (SIGFPE); divide->shift + 64B pad on the hot path; reuse raid_bdev_find_by_name - 0008 raid-rebuild: drop lock_range_cb function-pointer cast (compile-time safety) Verified: series applies cleanly in order; only the 4 intended patches changed; tier/cbt/rpc + bdev_raid_heat type-check against SPDK headers; host SB test passes.
Each test calls make_composite() which heap-allocates 3 tier_band nodes into t->bands but never freed them — 5 tests × 3 = 15 leaked nodes (3960 bytes). macOS ASan has no LeakSanitizer so it passed locally; the Linux CI (ASAN+LSAN) caught it. Add a free_composite() teardown and call it at the end of each test.
…cycle UAF hardening
Implements the fork-owned items of docs/audits/2026-07-04_plan-action-besoins-fork.md
(the confrontation of spdk-csi's activation needs against the fork). csi benches
(V-EC, substrate) and the semver release tag remain out of this repo.
P1.1 — EC targeted review (raid5f rebuild_ranges + degraded-read):
Verdict: the EC path is sound, not "written blind" — all parity math flows
through upstream read/write (0006 reuses raid5f geometry verbatim; 0008 reads
full stripes through the raid and writes them back, no hand-rolled parity).
Hardening (0008): raid_repair_read_done now names the unrecoverable chunk
(LBA/range) on a >1-fault stripe instead of an opaque -EIO.
Load-bearing finding: reconstruct-on-read only fires for a DEGRADED/absent
member, NOT a fresh zeroed replacement (reads its zeros successfully) — so the
control-plane must drive rebuild_ranges in remap→rebuild order, documented in
RPC-CONTRACT.md and docs/audits/2026-07-04_revue-ec-rebuild-ranges.md.
P1.2 — C3 geometry (0008, bdev_raid.c):
raid_bdev_write_info_json emits full_stripe_blocks = strip_size ×
min_base_bdevs_operational (the exact alignment rebuild_ranges validates),
removing the num_base_bdevs_operational(=n) vs min(=k) trap for csi.
P3.5 — SEC1 audit trail (new patch 0011 + wiring):
jsonrpc captures SO_PEERCRED at accept and exposes
spdk_jsonrpc_request_get_peer_ucred() + spdk_jsonrpc_request_audit(). Every
destructive handler logs "audit rpc=… peer=pid,uid,gid …":
tier delete/retire/resync (in-repo), lvol relocate/remap/batch (0005), nvmf
pause (0003), raid rebuild_ranges (0008). 0011 is a shared substrate applied
with the series (README documents the forward reference).
P3.7 — lifecycle UAF/race hardening (in-repo, deferred review items):
- tier SB fan-out: a bdev_tier_delete arriving mid-fan-out is deferred behind
it (delete_pending) — no more freeing `t`/closing base descs under in-flight
SB writes; tier_sb_fanout_complete reordered + vbdev_tier_sb_fanout_idle().
- tier hot-remove: a degraded band's channel-drain+close is deferred behind an
in-flight fan-out (close_pending) — honors the channel-before-desc contract.
- cbt rebuild: pin the cbt vbdev's lifetime for the whole rebuild (extra desc
when a source_bdev_name override means src_desc doesn't already pin it) +
abort on REMOVE — fixes the UAF where a base hot-remove freed the cbt node
(and its frozen bitmap/epoch) under a running rebuild's bare ctx->cbt.
These are async paths macOS can't run — validate on Linux ASAN / csi bench.
P3.8 — contract + decisions:
RPC-CONTRACT.md gains C3 (full_stripe_blocks + EC reconstruct precondition),
G3 (ENOSPC CAPACITY_EXCEEDED divergence: legs stay ONLINE, no auto-reconverge),
and the SEC1 audit note. D4/D5/D6 recorded in
docs/audits/2026-07-04_decisions-d4-d5-d6.md (D5: stay -m 0x1).
Patch series 10 → 11; patches.sh check: all 11 apply cleanly on pinned 2ef883e.
All edited in-repo + vendor files pass -fsyntax-only against the patched SPDK
headers (SO_PEERCRED capture is #ifdef __linux__).
tier_sb_fanout_complete now calls vbdev_tier_sb_fanout_idle (defined in vbdev_tier.c, not host-compilable). The T1 harness #includes vbdev_tier_sb.c but does not link vbdev_tier.c → undefined reference at link. Add a no-op stub (returns false = nothing deferred), matching the existing bdev-I/O stub pattern; the SB serialize/valid/select tests never drive a fan-out to completion.
Remediates the code-review findings (docs/audits/2026-07-04_plan-remediation-
revue-code-pr13.md) and treats the deferred items (…_fork-deferred-items…).
Tier module (module/bdev/tier):
- R2 register rehydrates t->seq from on-disk SBs → seq monotone across restart
(fork-owned; the control-plane never threads seq — contract already assumed it)
- R3 relocate flushes the destination unconditionally (durability), verify opt-in
- R4+R5 md-mirror divergence: per-leg md-ness from band + base offset (correct for
whole-device ops); degrade md legs on submission failure, not just completion
- R6 retired band = hole: FLUSH/UNMAP/WRITE_ZEROES skip it instead of -EIO
- R7 validate blocklen↔SB-slot early (reject T10-DIF 520/4160 at add/assemble)
- R8 assemble_band -EBUSY after register (band table frozen at register)
- R9 unified async_inflight quiesce-before-teardown (relocate/resync/rehydrate);
fanout_idle no longer drops a queued follow-up on delete_pending (would strand
a coalesced resync cb → leaked ref → hung delete; found in adversarial review)
- R13 reject cluster_blocks==1 (keep 0=legacy, ≥2=grain)
- R15 free req.name on decode error in 7 RPC handlers
- R17 reject md_num_blocks near UINT64_MAX (align overflow)
- #3 evariops_get_capabilities methods[] filtered through the live RPC registry
CBT module (module/bdev/cbt):
- R1 distinct CBT_REBUILD_ABORTED state; an aborted rebuild is not COMPLETED
- R10 throttle poller returns BUSY without dereferencing ctx after submit_next
- R12 documented non-bug (SPDK defers inline completions via in_submit_request)
Patches:
- R11 (0004/0005) remap quarantines the old cluster (release_old=false) instead of
freeing it to the thin pool → no re-serve into the dead band
- #5 (0004) F2 snapshot guard made intrinsic to spdk_blob_relocate_commit
- R16 (0005) decode_batch_items re-entry guard (duplicate JSON key leak)
- R14 (0010) umask(077) around bind() → atomic owner-only socket (closes TOCTOU)
Series regenerated via scripts/patches.sh; 11/11 apply cleanly on 2ef883e.
Deferred #1 (md-on-raid1) and #2 (heat passthru): documented decisions to keep the
current design (contract-stable, CSI-coupled; refactors not justified now). #4/#6
documented deferrals. L1/L2 left latent (neutralized by D5 -m 0x1).
Docs: RPC-CONTRACT.md + FORMAT-tier-superblock.md updated for the changed invariants.
Batch analogue of bdev_lvol_remap_cluster (patch 0005): collapse N
per-cluster remap round-trips into one RPC for a DEGRADED-band drain,
before a single bdev_raid_rebuild_ranges. No freeze / no copy / no
verify (the source band is dead — nothing to drain or read back).
- remap_one_precheck(): factor the per-item guards (cluster allocated ->
source DEGRADED -> claim dst -> dst ACTIVE); on success a claim is held,
on failure none. Single rpc_bdev_lvol_remap_cluster refactored to call
it, so single and batch apply provably identical guards (basis of the
idempotent-reissue property).
- remap_batch_ctx + async chain (open blob -> step -> commit(release_old=
false, R11 quarantine) -> next); sequential, first per-item error stops
the batch releasing its un-committed claim; returns {remapped, requested,
error}; partial success is a 200; bounded to RPC_BATCH_MAX (4096).
Dedicated decode_remap_batch_items (distinct container type, not a
CONTAINEROF type-pun onto the relocate struct). SEC1 audit hook.
- capabilities: add "bdev_lvol_remap_clusters" to g_evariops_methods[]
(runtime-filtered via spdk_rpc_get_method_state_mask).
- RPC-CONTRACT.md: contract entry with R11 quarantine + idempotency notes.
patches.sh verify: 11/11 apply cleanly (only 0005 changed). TU compiles
clean under SPDK's warning posture (mk/spdk.common.mk).
spdk_blob_claim_cluster_in_range probed the used_clusters bit-pool one BIT at a time from c_start on every call, so a campaign filling one band re-scanned its allocated prefix each time -> O(N^2) on the reactor (under the batch relocate's held freeze). Perf only; claim semantics unchanged. patch 0004 (the claim primitive): - spdk_bit_pool_find_first_free_from(pool, start): read-only accessor that delegates to the word-wise spdk_bit_array_find_first_clear, keeping the pool abstraction (bit_pool.h, bit_array.c, spdk_util.map). - blob_claim_cluster_in_range(..., cursor, ...) static core: word-wise scan (skip fully-allocated 64-bit words) + optional in/out cursor. spdk_blob_claim_cluster_in_range is now a cursor=NULL wrapper (existing callers unchanged, still get word-wise); new spdk_blob_claim_cluster_in_range_from exposes the cursor (blobstore.c, blob.h, spdk_blob.map). patch 0005 (thread the cursor through the batches): - struct claim_cursor + claim_cursor_for(): advance across consecutive same-window items, reset on window change so a stale hint can't spuriously -ENOSPC. Embedded in batch_ctx (relocate) and remap_batch_ctx (remap); remap_one_precheck gained a cursor param (NULL for the single form). Correctness: cursor==NULL is bit-identical to the old scan; the cursor is a pure hint (start=max(c_start,*cursor), results bounded to [c_start, c_end), set_bit_allocated re-checks), so a stale/over-shot cursor never yields an allocated or out-of-window cluster. Turns an N-claim campaign filling one band from O(N^2) into O(bandsize + N). patches.sh verify: 11/11 (only 0004+0005 changed). TUs compile clean under SPDK's warning posture. No new RPC (C-API only).
…antics) The live bitmap was never cleared during an epoch's lifetime: freeze memcpy'd it and rebuild finalize recomputed residual from the same accumulating bitmap. Iteration N therefore recopied the union of everything dirtied since epoch open — residual was monotonically non-decreasing and the control plane's iterate-until-converge loop could not terminate by design (observed on turing: 61-89% dirty at every freeze, full-size copies every iteration). Freeze now snapshots via atomic per-byte EXCHANGE and leaves the live bitmap empty, so each freeze yields the delta since the previous one and iterative partial rebuilds converge geometrically once copy rate exceeds dirty rate. Exchange (not memcpy+memset): a bit OR'd by an IO thread between copy and clear would be lost — a missed chunk under skip_rebuild is silent data divergence. Epochs share the single live bitmap, so the clear only happens when no OTHER epoch is active; otherwise the old snapshot-only behavior is kept (degraded convergence, still correct). Test mirrors updated to the new contract; new properties: sole-epoch freeze clears live / multi-epoch freeze preserves it / 200 re-freezes under a concurrent marker thread lose and invent zero bits (ASAN+UBSAN).
The post-write durability flush (F-6) EIO'd on every base bdev without FLUSH support — bdev_uring in particular — so NO tier superblock was ever persisted on such nodes: 'tier: superblock persist failed rc=-5' at every register/retire/relocate persist, and seq monotonicity never had an on-disk anchor. Same contract as the CBT rebuild path: a base without FLUSH has no volatile cache to drain — treat the completed write as durable and skip the flush. Mock header/harness extended accordingly.
Rolled nodes intermittently boot with 'Performing recovery on blobstore' (~70-95s md replay on SATA vs ~6s clean load) — the shutdown unload outcome decides it, and upstream only logs it at INFOLOG (invisible in production). Controlled experiments on turing: graceful pod delete unloads clean both idle and under active fio; only rollout-restarted nodes end up dirty, and not all of them — a shutdown race we cannot attribute without the errno. Patch 0012 logs all three outcomes (clean via fini_start, clean via unregister path, FAILED with errno) plus the fini_start skip at NOTICE. git-apply validated against v26.01 (2ef883e) after the existing series.
… lifecycle guards - H1: an exchanged-out frozen delta is merged back into the live bitmap before its buffer is discarded (re-freeze, close, evict) unless a rebuild COMPLETED — a failed/aborted iteration no longer silently loses chunks under skip_rebuild. Freeze now allocates before freeing (ENOMEM leaves the epoch intact). New cbt_epoch.frozen_live_consumed ownership flag. - H2: dirty bits are RE-marked at I/O completion (before the host ack), so a snapshot-and-clear freeze consuming the submit-time bit of an in-flight write is harmless — no drain around freeze required. Lock-free. - C3: epoch_invalidate refused (-EBUSY) while a rebuild is RUNNING, and the max-epochs eviction refuses epochs a rebuild ctx still points at (UAF). - M1: rebuild finish gate adds !aborted — a hot-remove abort classifies ABORTED (resumable), no longer FAILED via a doomed flush. - tests: resilience model aligned with production semantics (guards CBT-1/2, safe eviction) + 7 framing tests (H1 x4, H2, C3, M1). 60/60 green under ASan/UBSan. README rewritten to the reset-driven model (M4).
…semantics - C2: band drain no longer puts a reactor's base channel with legs in flight (per-reactor inflight[] counters, no atomics — completions land on the submitting thread; deferred put at the last completion via drain_refs) and no longer closes the desc under a relocate/resync engine (desc_pins; a close landing while pinned defers to the engine's terminal unpin). The final close runs on the app thread (spdk_bdev_close thread contract). - C4: md-resync re-validates src/dst liveness before every chunk, write and final flush (-ENODEV early abort); the C2 pins make a close under the engine structurally impossible. - H3: composite-global SB state (seq, pending cbs, inflight/queued flags) is owned by t->thread; the reactor-side md-leg degrade funnels its persist through spdk_thread_send_msg and re-resolves the composite via g_tier_nodes. - H5: shared tier_flush_or_durable() — no-FLUSH bases (bdev_uring) treat the completed write as durable instead of failing every relocate/md-resync with -ENOTSUP (same contract as the SB persist fix 54d02b2). - H6: a whole-device FLUSH no longer fails on a DEGRADED data band — the segment is a logged no-op (range already unreachable via -EIO on R/W); mutations and md legs keep failing honestly. Single-disk loss no longer turns every FS barrier into a full-volume read-only outage. - M2: the SB fan-out ctx-calloc ENOMEM path now runs vbdev_tier_sb_fanout_idle() — a deferred delete behind a queued follow-up is no longer stranded (composite/claims leak, -EEXIST on re-create). - M3: a fan-out that launched ZERO band writes completes -ENODEV, never rc==0 — the MJ6 durability contract is honored (a retire on a composite with all SB-carrying bands degraded is no longer acked as durable). - tests: test_tier_sb gains calloc fault injection + M2/M3 framing tests against the production code.
…004/0005) Regenerated mechanically via scripts/patches.sh regen from a 2ef883e clone with the series applied as commits (git am + rebase --autosquash) — the first fully mechanical round trip; patches.sh check passes 12/12 on a pristine pinned tree. - C1-DRAIN (0005): the relocate flows (single + batch) now take an LVOL-BDEV spdk_bdev_quiesce before the blob freeze: outstanding host I/O is DRAINED (LBA-range lock semantics) and new I/O is held ABOVE the L2P translation (re-translates on release). The blob freeze alone only gated new submissions — a write already in flight to the bs_dev could land on old_lba AFTER the copy read it: ACKed write lost on commit, cross-blob overwrite once old_lba was reallocated, undetectable by the C5 CRC. Release order: unfreeze → unquiesce (also on the leaked-freeze path). remap is unaffected (source band DEGRADED, no live-source copy). - H4 (0004+0005): an ambiguous commit failure (extent write dispatched, error completion — it may still have reached media) QUARANTINES the destination cluster instead of releasing it: a durable extent may reference it, and re-serving it would create two durable owners after restart. Bounded single-boot in-RAM leak; replay reconciles either way (contract documented in blob_relocate_extent_written). - Side effect of the mechanical regen: subjects normalized x/11 → x/12 and 0012's hand-edited hunk stats corrected; README table gains the missing 0012 row and the CFLAGS provenance fix (patch Makefile hunk, not Dockerfile sed) — audit A2.
- Breaking (D3/L1): bdev_cbt_epoch_list no longer emits healthy_clear_suspended / backends_healthy (dead state removed). - epoch_invalidate -EBUSY guard (C3) and the H1 delta-preservation invariant. - relocate section updated to the C1-DRAIN barrier (lvol-bdev quiesce + inner blob freeze) and the H4 quarantine contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SPEC-73 data-plane for storage tiering. Adds a
bdev_tiercomposite module + the lvol/blobrelocate primitives + nexus logical heat instrumentation, plus raid5f resilience hardening.
The out-of-tree SPDK changes ship as a 10-patch series (
patches/) applied onto the pinnedbase at image-build time;
scripts/patches.shmakes the round-trip mechanical.Modules / patches
bdev_tiercomposite (concat fast→slow bands + mirrored md region,per-band EIO isolation, on-disk superblock v2 à la bdev_raid_sb). RPCs: create/add_band/
assemble_band/register/retire_band/get_bands/resync_md/delete +
evariops_get_capabilities. (M1)release_cluster_lba) + public blob I/O freeze. (M2)
bdev_lvol_get_cluster_placement+bdev_lvol_relocate_cluster/ batch relocate(claim → freeze → copy → crash-safe L2P swap → unfreeze). (M2a/M2b)
double-fault guard (bail -EIO if a second member is unavailable). (M3 + hardening)
bdev_raid_rebuild_rangesread-reconstruct-writeback range repair. (B2/M4)Code-review remediation (this round)
A max-effort review surfaced ~31 findings (correctness first). The load-bearing ones are fixed;
architectural/policy items are called out below as deliberate follow-ups.
In-repo (
module/bdev/tier,module/bdev/cbt,scripts/):leg_iodouble-free on the md-read mirror-failover retry path.add_bandgains aTIER_MAX_BANDSbound +registeredguard (closes abase_ch[]out-of-bounds and a post-register geometry-divergence path).
io_channelinstead of leaving an ACTIVE band with no leg.dst_band->descacross the relocate copy chain (hot-remove mid-copy → clean EIO,not a NULL deref).
io_type_supportednow AND-consults the base bands for UNMAP/FLUSH/WRITE_ZEROES.assemble_band.100% FAILED) and any op crossing the md/data boundary; RESET now propagates to all base bands.
cluster_blockscap (v2 SB is u64); drop the deadcapacity_blocks/used_blocksfields.out_rebuild_idonstart_rebuilderror; the epoch-takeover (c5) guard now keys on anactually-running rebuild — fixes a permanent
-EBUSYafter a completed rebuild.patches.sh:checkis now non-destructive (validates against a scratch index, never touches thecaller's tree); NUL-safe patch enumeration.
Patch series (regenerated via
patches.sh regen; only the 4 touched patches changed):paused subsystem is destroyed).
blocklen == 0before dividing (SIGFPE on a not-yet-online nexus); hot-pathdivide→shift + 64-byte-padded region struct; reuse
raid_bdev_find_by_name.lock_range_cbfunction-pointer cast (a future internal-typedef change now breaks thebuild instead of corrupting the stack at runtime).
Deferred (design/policy, tracked for follow-up): compose the md mirror over raid1 instead of the
hand-rolled RAID1; heat as a passthru vbdev / per-channel counters; capabilities via the live RPC
registry; blob claim-scan word-wise + resume cursor; move relocate guards into blobstore; raid1
ENOSPC reconvergence policy.
Validation
lint(incl. T1 tier host unit test, ASAN+UBSAN+LSAN) ✓ and Build ✓ on debug+release× amd64+arm64 (real Linux SPDK build applying all 10 patches). Trivy fails as expected on feature
branches (image not pushed).
make_compositewere neverfreed — masked locally because macOS ASan has no LSan).
patches.sh checkconfirms the 10-patch series applies cleanly in order against a fresh pinned base.Label
build-imagesapplied to publish a prerelease image for downstream (spdk-csi) testing.