Fix generic CSI changeID retrieval and honor snapshot class deletion policy for CBT retention - #10307
Fix generic CSI changeID retrieval and honor snapshot class deletion policy for CBT retention#10307kaovilai wants to merge 3 commits into
Conversation
…policy Two defects in the CSI snapshot exposer, both of which make every CBT incremental silently degrade to a full backup on non-vSphere drivers. 1. getCBTInfo read the changeID from the freshly-created backup VSC's status, which is populated asynchronously and is therefore usually empty at that point. The handle is already present in the spec, so fall back to vsc.Spec.Source.SnapshotHandle. vSphere is unaffected -- it takes the VSphereCNSChangeIDAnno branch and never reads the VSC handle -- so this affects every other CSI driver. 2. createBackupVSC hardcoded DeletionPolicy: Delete, so the physical snapshot was removed when the backup completed. Case-2 storage such as Ceph RBD requires the base snapshot to survive for the next GetMetadataDelta call, and the block-data-mover design specifies a RetainSnapshot volume-policy parameter that was never implemented. Inherit the source snapshot class's deletion policy instead, and clean up the backup VSC object in CleanUp. Adds TestCreateBackupVSCDeletionPolicy covering both Delete and Retain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> (cherry picked from commit f4867d0) Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
✅ Deploy Preview for velero canceled.
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
For a Case 2 driver (design/block-data-mover/block-data-mover.md), such as Ceph RBD, rbd snap diff needs the base and target snapshots in the same clone chain. Delete destroys the base as soon as the backup completes, so the next incremental's delta query fails and degrades to an allocated-blocks backup (or a full whole-device transfer without that fix). Inheriting Retain there isn't an optional nicety, it's what makes incrementals possible at all. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
@sseago @shubham-pampattiwar 0/ please review |
| // The backup VSC is statically provisioned from the source VSC's | ||
| // snapshot handle; its status is populated asynchronously and may | ||
| // not be set yet, but the handle is already in the spec. | ||
| cbtInfo.changeID = *vsc.Spec.Source.SnapshotHandle |
There was a problem hiding this comment.
The changeID fallback here is the core fix for #10294, but it isn't covered by a unit test. The existing TestGetCBTInfo case "fallback to pv and vsc snapshot handle" sets Status.SnapshotHandle, which exercises the old path, not this new else if. Codecov's two uncovered lines are this branch plus the new CleanUp deletion.
Could you add a TestGetCBTInfo case with vsc.Status = nil and vsc.Spec.Source.SnapshotHandle set, asserting changeID comes from the spec handle? That locks in the exact regression #10294 was about, so a future refactor of getCBTInfo can't silently reintroduce the empty changeID.
| cbtInfo.changeID = *vsc.Status.SnapshotHandle | ||
| } else if vsc.Spec.Source.SnapshotHandle != nil { | ||
| // The backup VSC is statically provisioned from the source VSC's | ||
| // snapshot handle; its status is populated asynchronously and may |
There was a problem hiding this comment.
Small wording nit: this comment says status "is populated asynchronously and may not be set yet," which reads like a timing issue that a re-fetch would resolve. But this vsc is the in-memory object returned from Create() and never re-fetched, so its status is unconditionally nil here, not just temporarily. The description puts it well ("nil unconditionally, not merely until some async fill completes"). Matching the comment to that avoids someone later "fixing" this with a re-fetch that wouldn't help.
|
The description's "Minor accompanying fix" for the Could you either pull the one-liner in (have |
Note
Responses generated with Claude
Does your change fix a particular issue?
Fixes #10294
Also fixes the snapshot-retention gap discussed on #9714 (
RetainSnapshotvolume-policy parameter from the design doc, never implemented) — no separate issue number, since that discussion lives on #9714 itself.Summary
Two independent fixes to the CSI snapshot exposer, landed together since they touch the same file and were both needed to get a working Ceph incremental end-to-end:
1. Generic changeID retrieval (#10294).
createBackupVSCconstructs the backup VolumeSnapshotContent with onlyObjectMetaandSpec— it never setsStatus— and returns the object straight fromCreate(). Sovsc.Statusis nil unconditionally, not merely until some async fill completes; nothing re-fetches it beforegetCBTInforeads it. The generic branch (every driver except vSphere, which takes the annotation branch instead) therefore gets an empty changeID on 100% of runs. Fix: readbackupVSC.Spec.Source.SnapshotHandle(set at creation from the ready source VSC'sStatus.SnapshotHandle) as a fallback whenStatusisn't populated.2. Snapshot retention / Case-2 support (design doc lines 362-374 on #9714). The backup VSC's
DeletionPolicywas hardcoded toDelete, so any driver requiring the base snapshot to persist forGetMetadataDelta(Ceph RBD — "Case 2" in the design doc) had its base snapshot deleted immediately after upload, making every incremental fall back to full. Fix: honor the source VSC's (or its VolumeSnapshotClass's) ownDeletionPolicyinstead of hardcodingDelete.Minor accompanying fix: the
retained=log field was(retained != nil), butRetainVSCreturns non-nil on both the already-Retainand just-patched branches, so the field was alwaystrueregardless of whether a patch actually occurred. Now reflects whether a patch happened.Measured before/after (live Ceph/ODF validation)
Before: every incremental on Ceph RBD fell back to a full transfer (100% of device size) because the base snapshot was gone by the time
GetMetadataDeltaran.After: an incremental with a 20 MiB delta moved 20,971,520 B — exactly the written delta, not the device size. Restore verified byte-for-byte against source (all checksums matched).
Live validation
Both fixes here were developed and validated live on a real Ceph/ODF cluster as part of a combined branch carrying all five fixes from this campaign together, since this changeID fix is the prerequisite that makes every other fix's incremental-path testing possible — full diff: main...kaovilai:velero:ceph-changeid
SnapshotMetadataServiceCRD + upstream sidecar v1.1.0): changeID round-trips correctly through a full→incremental pair; base snapshot survives whenDeletionPolicy: Retain; incremental delta is exact; restore is byte-identical.retained=log field correctness (not independently exercised live, verified by code inspection and the existing test fixture).What was deliberately not changed
This surfaces a consequence that is intentionally not fixed here: once base snapshots are retained, nothing currently reclaims them (unbounded orphan growth). That's tracked as #9835.
Tests
TestCreateBackupVSCDeletionPolicy(table test,DeleteandRetainsubtests, both pass). Note for reviewers: the pre-existing assertion this replaces/extends atcsi_snapshot_test.gowas vacuous — its fixture policy was alreadyDelete, so it passed against the old hardcoded value regardless of whether inheritance worked.Please indicate you have done the following:
make new-changelog) — will run once this PR is open.site/content/docs/main— n/a, no new user-facing flag; behavior fix to an existing feature.