miroir testing

[onedr0p]

Prompt:
mirior was update to 0.11.3 - Review miroir status, check logs, metrics, events and make sure everything is okay. next run a common fio test for testing drives on the SC and report back the findings. after, if you can test the new PVC clone and volumesnapshot features

Validation of the miroir chart upgrade to 0.11.3 on the home-ops cluster: post-upgrade health review, fio benchmarks on the miroir-slow StorageClass, and functional testing of the PVC clone and VolumeSnapshot features.

Upgrade health

• HelmRelease upgraded successfully (miroir-system/miroir.v6, chart 0.11.3); all 3 agent pods and both controller replicas rolled cleanly and are Running with no restarts.
• All 19 MiroirVolumes report Ready 2/2, all peers connected, splitBrain: false everywhere.
• k8s-0 shows Diskless on every volume. This is expected: with replicas: "2" the data lives on k8s-1/k8s-2 and k8s-0 attaches as the quorum tiebreaker (its ALLOCATED column is empty for the same reason).
• Controller and agent logs since the rollout contain zero errors or warnings. Namespace events are all normal rollout/leader-election activity; both Flux kustomizations (miroir, miroir-config) reconciled successfully.
• Metrics (agent :9810/metrics): DRBD kernel 9.3.2 / utils 9.34.3, miroir_volume_diskless_primary is 0 for every volume, pool meta usage ~1.9%, ~10 GB allocated of 998 GB per data node.
• The miroirsnapshotgroups CRD is now present (created 2026-07-18), so group snapshots are unblocked.

fio benchmarks (miroir-slow)

Setup: 5Gi PVC on miroir-slow (replicas=2, ext4), alpine pod, fio with --ioengine=libaio --direct=1 --time_based --runtime=30 --numjobs=1 against a 2G file.

Test	Result	Avg latency
4k randread, QD32	222k IOPS (867 MiB/s)	144 µs
4k randwrite, QD32	35.0k IOPS (137 MiB/s)	913 µs
1M seq read, QD16	2837 MiB/s	5.6 ms
1M seq write, QD16	834 MiB/s	19 ms

Notes:

• Reads are served from the local replica, so they run at effectively local NVMe speed.
• Writes carry the expected DRBD synchronous replication penalty (every write commits on both k8s-1 and k8s-2 over the network). Write tails were tight: 99%+ of 4k writes completed under 2 ms; seq write p99 was in the 50 ms bucket with a 119 ms max.
• No fio errors in any run.

PVC clone and VolumeSnapshot testing

Method: seeded the source PVC with a 64 MB /dev/urandom file and recorded its sha256, then exercised both CSI data sources and verified integrity.

VolumeSnapshot (class miroir):

• VolumeSnapshot went ReadyToUse in ~2 s; the backing MiroirSnapshot CR reported READY true against the source volume.
• A PVC restored from the snapshot (dataSource: VolumeSnapshot) provisioned and bound normally.

PVC clone:

• A PVC with dataSource: {kind: PersistentVolumeClaim} provisioned and bound normally.

Verification:

• Both the clone and the restore mounted with the full filesystem contents intact (fio test file, seeded file, checksum file).
• sha256 of the seeded file matched the source exactly on both volumes:
  • expected: 0602001021504a1c26f50e4e204953fb93341a137342bfbcbb1058b99f38f0ce
  • clone: match
  • restore: match
• Both volumes were writable after provisioning.
• Immediately after creation both new volumes showed phase Creating 0/2 with the second replica Inconsistent while the background resync ran; both converged to Ready 2/2 in under a minute (~2.2 GB of data). Judge readiness by perNode diskState rather than phase during this window.

Cleanup

The test namespace was deleted afterward; confirmed no leftover PVs, MiroirVolumes, or MiroirSnapshots.

Verdict

No issues found with 0.11.3. Health, logs, metrics, benchmarks, and the clone/snapshot features all check out.

[onedr0p]

Future Prompt:

Prompt:
mirior was updated - Review miroir status, check logs, metrics, events and make sure everything is okay. next run a common fio test for testing drives on the SC and report back the findings make sure to do this for pods where the PVC is local to the node and another test over the network. after, test the new PVC clone and volumesnapshot and volumesnapshot group features

[onedr0p]

Re-validation on miroir 0.11.8

Follow-up to the 0.11.3 validation above. Chart was upgraded 0.11.7 ➔ 0.11.8 (HelmRelease miroir-system/miroir.v11, deployed 2026-07-22 22:09 UTC). This run adds a local-vs-network fio comparison and exercises PVC clone, VolumeSnapshot, and VolumeGroupSnapshot.

Upgrade health

• HelmRelease Ready=True, chart miroir@0.11.8 (appVersion 0.11.8, commit fd8ad39). All 3 agent pods (2/2) and both controller replicas (4/4) Running, 0 restarts.
• All 19 MiroirVolumes report Ready 2/2. DiskState across the fleet: 38 UpToDate + 19 Diskless (k8s-0 tiebreaker), 0 split-brain, 0 disconnected.
• Metrics (:9810): DRBD kernel 9.3.2 / utils 9.34.3, miroir_volume_split_brain sum 0, miroir_volume_connected 19/19, no diskless_primary, conversion_webhook_panics 0. Pool slow: 21.66 GB allocated of 998 GB, meta used 2.59%.
• Controller + agent logs since the rollout: zero errors/warnings. Namespace events are normal rollout/reconcile; both Flux kustomizations (miroir, miroir-config) reconciled clean; OCIRepository artifact up-to-date.

fio benchmarks — local vs over-the-network

Same method as before: 5Gi PVC on miroir-slow (replicas=2, ext4), --ioengine=libaio --direct=1 --numjobs=1 --time_based --runtime=30 --ramp_time=5 against a pre-laid-down 2G file.

To isolate the network path I pinned two pods to specific nodes and confirmed the DRBD role of each before testing:

• Local = pod on a node holding a diskful replica → diskState: UpToDate (co-located primary).
• Network = pod on the volume's diskless node → diskState: Diskless (diskless primary; every read and write traverses the network to the two diskful peers).

Test	Local (diskful primary)	Network (diskless primary)	Δ
4k randread, QD32	228k IOPS (890 MiB/s), 136 µs	117k IOPS (456 MiB/s), 270 µs	~0.51×
4k randwrite, QD32	34.9k IOPS (136 MiB/s), 908 µs	32.1k IOPS (125 MiB/s), 994 µs	~0.92×
1M seq read, QD16	2842 MiB/s, 5.6 ms	1090 MiB/s, 14.6 ms	~0.38×
1M seq write, QD16	840 MiB/s, 18.9 ms	567 MiB/s, 28.1 ms	~0.68×

(latency = avg completion latency; p99 randread 268 µs local / 544 µs network, p99 randwrite 1.29 ms / 1.76 ms.)

Reads show the expected split: the co-located primary serves reads from its local NVMe leg, while the diskless primary pulls every block over the network (~2× randread, ~2.6× seq read). Writes are close because DRBD replicates synchronously either way — the diskful primary already commits every write to a remote peer, so it's network-bound to begin with; the diskless primary is only modestly slower (both legs remote, no local commit), with the gap widening on the bandwidth-bound seq-write (two remote copies). No fio errors in any run. Local numbers are consistent with the 0.11.3 baseline.

PVC clone + VolumeSnapshot — ✅ pass

Seeded a source PVC with a 64 MB /dev/urandom file (sha256sum recorded), then exercised both CSI data sources and verified with sha256sum -c:

• VolumeSnapshot (class miroir): ReadyToUse in ~2 s; backing MiroirSnapshot READY true. PVC restored from it bound and mounted with contents intact.
• PVC clone (dataSource: PersistentVolumeClaim): bound and mounted; created a backing clone-* MiroirSnapshot under the hood.
• Both sha256sum -c seed.sha256 → /data/seed: OK. Data integrity confirmed on both.

VolumeGroupSnapshot — ⚠️ does not complete (config gap)

Created a VolumeGroupSnapshotClass (driver miroir.home-operations.com) and a VolumeGroupSnapshot selecting two labeled miroir-slow PVCs. It never becomes ready: the snapshot-controller binds a VolumeGroupSnapshotContent, emits CreatingGroupSnapshot, and then waits indefinitely (readyToUse: false, content status: null, no member VolumeSnapshots, no MiroirSnapshotGroup CR created — vs ~2 s for an identical single snapshot).

Root cause is a sidecar config gap, not the driver:

• The bundled csi-snapshotter (v8.6.0) sidecar on the miroir controller runs with only --csi-address --timeout --leader-election. In v8.6.0 group snapshots are gated behind --feature-gates=CSIVolumeGroupSnapshot=true (BETA, default false), which the sidecar does not set, so it never reconciles the VolumeGroupSnapshotContent.
• The cluster-wide snapshot-controller does have CSIVolumeGroupSnapshot=true, which is why the VGS/VGSContent get created — but the request stalls at the miroir sidecar.
• The miroirsnapshotgroups CRD documents its spec as "created by the controller on CSI CreateVolumeGroupSnapshot", i.e. the k8s VolumeGroupSnapshot API is the intended entry point and the driver-side plumbing likely exists — it's just unreachable until the sidecar enables the feature gate.

Suggested fix: add --feature-gates=CSIVolumeGroupSnapshot=true to the chart's csi-snapshotter args (ideally behind a values toggle). Happy to test again once that's exposed.

Cleanup

Test namespace deleted; verified back to 19 MiroirVolumes with 0 MiroirSnapshots / MiroirSnapshotGroups / VolumeGroupSnapshotContents and no orphaned PVs.

Verdict

0.11.8 health, logs, metrics, benchmarks, PVC clone, and VolumeSnapshot all check out. The only issue is VolumeGroupSnapshot, which can't complete because the chart's csi-snapshotter sidecar doesn't enable the CSIVolumeGroupSnapshot feature gate.

[onedr0p]

miroir 0.11.9 — post-upgrade validation

Follow-up to the 0.11.3 run above, re-done after the bump to 0.11.9 (chart miroir@0.11.9, HelmRelease miroir.v13). TL;DR: healthy, no regressions. Clone / VolumeSnapshot / VolumeGroupSnapshot all pass with data integrity verified by sha256.

Health

• Controllers 4/4 Running (×2), agents 2/2 Running (×3), 0 restarts since the rollout.
• HelmRelease Ready=True — Helm upgrade succeeded for release miroir-system/miroir.v13.
• Prometheus: all 5 miroir targets up, no miroir/DRBD alerts firing.
• DRBD health, before and after the whole test workload: split_brain=0, disk_failed=0, out_of_sync_bytes=0, suspended=0; every replica series quorum=1 / connected / up_to_date.
• 19 existing volumes Ready 2/2. DRBD 9.3.2 / utils 9.34.3 on all nodes. Thin-pool metadata 1.9–2.7%, pool ~2.4% used.
• Controller + agent logs clean (no errors/warnings).

fio — local vs. over the network

fio 3.13, 4 GiB file, direct=1, libaio, 30 s/pattern (seq bs=1M iodepth=16, rand bs=4k iodepth=64). Three pod/replica placements on the miroir StorageClasses:

• Local — miroir-slow-local (replicas=1), pod co-located with its only replica → all I/O local.
• Replicated — miroir-slow (replicas=2), pod on a diskful node → reads local, writes sync-replicated to the peer over the network.
• Diskless — miroir-slow, pod scheduled on the node without a replica → all I/O over the network.

Pattern (metric)	Local	Replicated (writes over net)	Diskless (all over net)
Seq read — BW	2845 MiB/s	2844 MiB/s	806 MiB/s
Seq write — BW	4129 MiB/s	741 MiB/s	578 MiB/s
Rand read 4k — IOPS (p99)	252.8k (0.40 ms)	227.4k (0.52 ms)	103.3k (1.66 ms)
Rand write 4k — IOPS (p99)	188.8k (0.76 ms)	57.1k (3.49 ms)	53.9k (4.49 ms)

Takeaways:

• Reads from a local replica run at local speed — replication adds ~0 read cost (randread 227k vs 252k IOPS; seqread identical at 2.8 GiB/s).
• Synchronous replication is the write cost: seq write 4129 → 741 MiB/s, randwrite 189k → 57k IOPS, p99 0.76 → 3.5 ms. Expected for sync DRBD replication (writes wait on peer ack).
• Going fully diskless adds the read penalty (randread 103k, seqread 806 MiB/s) since reads now traverse the network too. The replication network is not the bottleneck — remote random-read still sustains 403 MiB/s / 103k IOPS.

Features (data integrity verified by sha256)

• PVC clone (miroir-slow): payload: OK, marker.txt: OK.
• VolumeSnapshot: readyToUse in ~2 s; restored PVC → payload: OK, marker.txt: OK.
• VolumeGroupSnapshot (new feature): group over 2 label-selected PVCs → readyToUse in ~13 s, fanned out to one member VolumeSnapshot per PVC; both restored with the correct per-volume data (OK / OK). Underlying miroirsnapshotgroup CR Ready.

Notes / follow-ups

1. No VolumeGroupSnapshotClass ships with the config. The CSIVolumeGroupSnapshot gate + groupSnapshots.enabled: true are on and the csi-snapshotter v8.6.0 sidecar is wired, but there's no VolumeGroupSnapshotClass in the cluster/repo — so group snapshots can't be used out of the box. I created a temporary one (driver: miroir.home-operations.com, deletionPolicy: Delete) to run this test and removed it after. Worth adding one alongside the existing VolumeSnapshotClass under kubernetes/apps/miroir-system/miroir/config/.
2. Group-snapshot creation is async and log-noisy. During the ~13 s convergence the driver returns CSI Aborted: "group snapshot … is not ready" on each poll and the external-snapshotter retries until ready — both sides emit error-level log lines even though it succeeds. The single-snapshot path doesn't (~2 s, clean). Cosmetic, but a candidate for an upstream log-level/backoff tweak.

Housekeeping

All test resources ran in a throwaway miroir-bench namespace and were removed — cluster is back to baseline (19 volumes, 0 snapshots). No repo changes.

_{fio image: docker.io/xridge/fio (3.13). Placement forced via nodeSelector + WaitForFirstConsumer; the "diskless" run used miroir-slow's auto-added diskless/quorum node.}

[onedr0p]

Opened #11371 to add the missing VolumeGroupSnapshotClass (follow-up #1 above).

[onedr0p]

mirior was updated - Review miroir status, check logs, metrics, events and make sure everything is okay. next run a common fio test using official debian trixie slim for testing drives on the SC and report back the findings make sure to do this for pods where the PVC is local to the node and another test over the network. after, test the new PVC clone and volumesnapshot and volumesnapshot group features, update the issue #11322 with the findings

[onedr0p]

miroir 0.11.10 — post-upgrade validation

Re-run after the bump to 0.11.10 (chart miroir@0.11.10, HelmRelease miroir.v14). This round the fio load ran from the official debian:trixie-slim image (apt-get install fio → fio 3.39) instead of the previous alpine image. TL;DR: healthy, no regressions, and the group-snapshot log noise flagged in the 0.11.9 run is gone.

Health

• Controllers 4/4 Running (×2), agents 2/2 Running (×3), 0 restarts since the rollout. HelmRelease Ready=True (v14).
• Prometheus: all 5 miroir targets up, no miroir/DRBD alerts firing.
• DRBD health, before and after the test workload: split_brain=0, disk_failed=0, out_of_sync_bytes=0, suspended=0; every replica series quorum=1 / connected / up_to_date.
• 19 existing volumes Ready 2/2. DRBD 9.3.2 / utils 9.34.3 on all nodes. Thin-pool metadata ~2.4%. Controller + agent logs clean.

fio — local vs. over the network

debian:trixie-slim + fio 3.39, 4 GiB file, direct=1, libaio, 30 s/pattern (seq bs=1M iodepth=16, rand bs=4k iodepth=64). Three pod/replica placements on the miroir StorageClasses:

• Local — miroir-slow-local (replicas=1), pod co-located with its only replica → all I/O local.
• Replicated — miroir-slow (replicas=2), pod on a diskful node → reads local, writes sync-replicated to the peer over the net.
• Diskless — miroir-slow, pod on the node without a replica → all I/O over the net.

Pattern (metric)	Local	Replicated (writes over net)	Diskless (all over net)
Seq read — BW	2843 MiB/s	2844 MiB/s	1146 MiB/s
Seq write — BW	3992 MiB/s	815 MiB/s	578 MiB/s
Rand read 4k — IOPS (p99)	230.2k (0.49 ms)	201.8k (0.61 ms)	162.0k (0.85 ms)
Rand write 4k — IOPS (p99)	168.7k (0.89 ms)	52.8k (3.92 ms)	53.8k (3.92 ms)

Same shape as the 0.11.9 run — no regression:

• Reads from a local replica run at local speed (randread 202k vs 230k; seqread identical at 2.8 GiB/s) — replication adds ~0 read cost.
• Synchronous replication is the write cost: seq write 3992 → 815 MiB/s, randwrite 169k → 53k IOPS, p99 0.89 → 3.9 ms.
• Diskless adds the read penalty (randread 162k, seqread 1146 MiB/s) since reads traverse the network. Network path is healthy — remote random-read still sustains 633 MiB/s / 162k IOPS.

Features (data integrity verified by sha256)

• PVC clone (miroir-slow): payload: OK, marker.txt: OK.
• VolumeSnapshot: readyToUse in ~2 s; restore → OK / OK.
• VolumeGroupSnapshot: now uses the VolumeGroupSnapshotClass added in feat(miroir): add VolumeGroupSnapshotClass #11371 (Flux-managed, kustomize.toolkit.fluxcd.io/name=miroir-config) — no temporary class needed this time. Group over 2 label-selected PVCs → readyToUse in ~12 s, fanned out to one member VolumeSnapshot per PVC; both restored with correct per-volume data (OK / OK).

Change since 0.11.9

• ✅ Group-snapshot log noise is fixed. The 0.11.9 run logged transient CSI Aborted: "group snapshot … is not ready" error lines on the controller + csi-snapshotter throughout the ~13 s convergence. On 0.11.10 the same test produced 0 such lines on either side — creation is still async (~12 s) but now clean.

Housekeeping

Test resources ran in a throwaway miroir-bench namespace and were removed — back to baseline (19 volumes, 0 snapshots). The Flux-managed VolumeGroupSnapshotClass was left in place. No repo changes.

[onedr0p]

miroir 0.11.14 validation

Upgraded via #11426 (0.11.13 ➔ 0.11.14). Health, fio local-vs-network, and clone/snapshot/group-snapshot all checked.

1. Health

Check	Result
miroir-controller	2 × 4/4 Running, 0 restarts
miroir-agent	3 × 2/2 Running, 0 restarts
HelmRelease	Ready, miroir-system/miroir.v18 chart miroir@0.11.14
Rollout events	clean, Health check passed in 45.0s
min	k8s-0/1/2 all present, DRBD 9.3.3, pool slow
ming	default = ["k8s-0","k8s-1","k8s-2"]
Controller/agent logs	0 error/warn lines across the rollout window
Prometheus targets	up{namespace="miroir-system"} = 5/5
Firing alerts	none from miroir

DRBD health metrics, 21 volumes × 2 replicas = 42 series, before and after the test run:

miroir_volume_split_brain       sum=0
miroir_volume_disk_failed       sum=0
miroir_volume_out_of_sync_bytes sum=0
miroir_volume_suspended         sum=0
miroir_volume_quorum            sum=42 / count=42
miroir_volume_connected         sum=42 / count=42
miroir_volume_up_to_date        sum=42 / count=42

2. fio: local vs over-the-network

debian:trixie-slim + apt-get install fio (fio 3.39), 20Gi PVC, --direct=1 --ioengine=libaio --size=4G --runtime=30 --ramp_time=5 --time_based. Pods pinned with nodeSelector. Three placements:

• Local - miroir-slow-local (1 replica), replica and pod both on k8s-1.
• Replicated - miroir-slow (2 replicas: k8s-1 + k8s-0, diskless quorum on k8s-2), pod on k8s-1, so reads are local and writes replicate over the net.
• Diskless - same miroir-slow PVC, pod moved to k8s-2 which holds no replica, so all I/O crosses the net.

Workload	Local	Replicated	Diskless
seq write 1M QD8	4187 MiB/s	839 MiB/s (-80%)	573 MiB/s (-86%)
seq read 1M QD8	4864 MiB/s	4869 MiB/s (+0%)	776 MiB/s (-84%)
rand write 4k QD32×4	491,672 IOPS	84,255 IOPS (-83%)	60,213 IOPS (-88%)
rand read 4k QD32×4	612,996 IOPS	587,520 IOPS (-4%)	123,306 IOPS (-80%)
rand write 4k QD1 (avg lat)	7 µs	784 µs (112×)	833 µs (119×)
rand read 4k QD1 (avg lat)	47 µs	47 µs (1.0×)	151 µs (3.2×)

Findings:

1. Reads are free when a local replica exists. Replicated vs local is within noise on every read metric (4869 vs 4864 MiB/s seq, 47 vs 47 µs QD1). DRBD serves reads from the local replica and never pays a network hop.
2. Replication costs ~80% of write throughput and dominates QD1 write latency (7 µs ➔ 784 µs). That is DRBD protocol C waiting on the peer ack.
3. Replicated writes are network-bandwidth-bound, not CPU-bound. 839 MiB/s ≈ 7.0 Gbit/s on a 10 GbE link (node_network_speed_bytes = 1.25 GB/s on bond0), so the write path is running near the practical ceiling of the interconnect rather than hitting a miroir limit.
4. Diskless placement is the case worth avoiding. Writes are barely worse than replicated (already net-bound), but reads collapse: seq read -84%, rand read -80%, QD1 read latency 3.2×. Keeping workloads scheduled onto a node that holds a replica is worth roughly 5× read throughput.

Caveat on the local QD1 write number: direct=1 bypasses page cache but does not force a flush, so 7 µs is the NVMe write-cache ack, not a durable-to-media latency. The comparison across the three placements is still apples-to-apples since all three use the same backing pool.

3. Features

All verified by sha256 against 512 MiB of /dev/urandom written to the source PVCs, source unmounted before snapshotting.

Feature	Result	Time to readyToUse
PVC clone (dataSource: PersistentVolumeClaim)	✅ sha256 match	bound on first consumer
VolumeSnapshot + restore	✅ sha256 match	~4s
VolumeGroupSnapshot (2 PVCs by label selector) + restore	✅ both members sha256 match	~9s / ~12s

source    a=1a9ed4ab…dfeb03   b=64be8810…c668de
clone-a   1a9ed4ab…dfeb03  ✅
restore-a 1a9ed4ab…dfeb03  ✅
vgs-a     1a9ed4ab…dfeb03  ✅
vgs-b     64be8810…c668de  ✅

The group snapshot correctly resolved the bench.group=grouptest label selector to both PVCs and emitted one member VolumeSnapshot each, both independently restorable.

4. Group snapshot logs an intentional retry signal as an error

Corrected after filing upstream: this section originally called the behaviour a 0.11.14 regression. That was wrong. internal/csi/groupcontroller.go and internal/csi/server.go are byte-identical between 0.11.10 and 0.11.14; the Aborted return has existed since 0.11.8 and the log interceptor since 0.11.7. Whether it appears is a race, not a code change — see below. Filed upstream as home-operations/miroir#379.

Reproduced on two consecutive group snapshots:

Run	Error lines	Window	Outcome
1	4	1.02s	ready at ~9s
2	4	1.02s	ready at ~12s

{"level":"error","logger":"csi","msg":"rpc failed",
 "method":"/csi.v1.GroupController/CreateVolumeGroupSnapshot",
 "error":"rpc error: code = Aborted desc = group snapshot groupsnapshot-<uid> is not ready", ...}

It also surfaces two Warning events on the VolumeGroupSnapshotContent:

• GroupSnapshotCreationFailed
• GroupSnapshotContentCheckandUpdateFailed

Functionally harmless: csi-snapshotter retries and the group snapshot completes with correct data every time.

The Aborted return at groupcontroller.go:145 is deliberate and contract-tested (groupcontroller_test.go:69) — it is a "call me again" signal to csi-snapshotter, used because external-snapshotter only records member readiness from a successful Create response. The problem is that logInterceptor (server.go:117-125) logs every non-nil error at error level with a stacktrace, with no status-code classification, so the one path that uses an error as flow control always looks like a failure.

Why 0.11.10 looked clean: if the round happens to be ready by the time the first CreateVolumeGroupSnapshot call evaluates result.GetReadyToUse(), line 145 never fires. That is what I hit on 0.11.10 and wrongly recorded as a fix. It reproduces reliably whenever the group needs more than one call to settle.

Suggested fix (in the upstream issue): classify by gRPC code in logInterceptor, logging Aborted/Unavailable/DeadlineExceeded at V(1) and reserving error+stacktrace for Internal/Unknown/DataLoss. That clears the log noise; the Warning events would remain, since csi-snapshotter emits those purely because the RPC returned an error.

5. Cleanup

Everything ran in a throwaway miroir-bench namespace. After deletion miv returned 29 ➔ 21 (baseline), no orphaned volumesnapshotcontent / volumegroupsnapshotcontent, and all miroir pods remain at 0 restarts.