Large VM Warm Migration failed - Failed to zero thick target #445
Replies: 12 comments 1 reply
-
|
Hi @neilcolyn-jpg, thanks for the detailed report and the full log, it made this quick to pin down. Root cause. Before the warm copy, ProxCenter zeroes each freshly-allocated disk on a thick LVM target so that any region the CBT copy does not touch reads back as zero (otherwise an unwritten area could expose stale blocks). On a thick target, the fallback that streams zeros with The fix (merged).
It will ship in the next release. Once you are on that build, re-running the same warm migration should get past the disk-1 zeroing, and I will follow up here when the release is out. To be clear, there was no data-integrity problem here: the disks were being zeroed correctly, the step was just mis-reporting a success as a failure. Fixed in #455. |
Beta Was this translation helpful? Give feedback.
-
|
Attempted the same migration again after updating to 1.4.4, migration went further, looks like it failed right at the end? [13:24:20] Warm migration: planning |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the update, and there is good news in here: the v1.4.4 zero-fill fix worked. All nine disks zeroed cleanly and the full copy moved the entire 1.43 TB, so the original problem is resolved. The new failure is at a later stage. After the full copy, ProxCenter runs a delta pass, and the first delta on disk 0 failed with To pinpoint it, could you grab a few logs from the node and the ProxCenter host, scoped to the time the migration failed (the run that ended around 17:29 with the EOF)? Please adjust the date and window to match the actual run. On the target Proxmox node (ld6-pve-node01): On the host running the ProxCenter containers: One more question: is there any firewall, NAT, or IDS/IPS between the ProxCenter host and the Proxmox node that could reset or rate-limit SSH connections? A warm migration opens a lot of short-lived SSH sessions, so a connection rate limit is one of the things we want to rule out. In parallel, we are adding extra logging around the delta-apply step in the next build so the exact cause is captured automatically on a re-run. Thanks again for the detailed reports, they have been very helpful. |
Beta Was this translation helpful? Give feedback.
-
|
Proxcenter Docker Logs.txt I attached the Proxcenter docker logs by running this command instead: docker logs --since "2026-06-22T17:24:00" --until "2026-06-22T17:35:00" proxcenter-orchestrator 2>&1 =============================== journalctl -u ssh --since "2026-06-22 17:00" --until "2026-06-22 17:45" --no-pager Returned no errors =============================== root@ld6-pve-node01:~# journalctl -k --since "2026-06-22 17:00" --until "2026-06-22 17:45" --no-pager | grep -iE 'oom|killed process|out of memory|nbdkit' Returned nothing ================================ root@ld6-pve-node01:~# sshd -T | grep -iE 'maxstartups|maxsessions|clientalive|logingrace' logingracetime 120 maxsessions 10 clientaliveinterval 0 clientalivecountmax 3 maxstartups 10:30:100 persourcemaxstartups none root@ld6-pve-node01:~# ======================= root@ld6-pve-node01: ======================= Proxcenter is in JH1 site and VMware/Proxmox nodes are in LD6 site. JH1 and LD6 sites are connected via IPsec on the firewalls. I have asked our network team to check if there are any "reset or rate-limit SSH connection" configuration that might cause this. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Neil, those captures nailed it. Good news: this is on our side, not your network, and we have a fix. Root cause. After the full copy, the delta pass writes each disk's changed blocks back to the target. We were batching every changed block of a disk into one SSH command. On disk 0 (your OS disk, which took the most writes during the multi-hour copy) that single command reached about 158 KB. Linux refuses to exec a command whose single argument is larger than 128 KiB (the MAX_ARG_STRLEN limit, which is separate from the much larger total argument limit), so the node rejected the command before any block was written and the SSH channel dropped before returning a status. That is exactly why you got an opaque "command failed: EOF" with no underlying error: nothing actually ran, which is also why every node-side check you ran came back clean. It only hit disk 0, and only at the delta stage, for the same reason: it is the busiest disk, so it was the only one whose batched command crossed the limit, and it runs first. The fix splits the delta apply into several smaller commands, each well under the limit, run in order. It is up for review now (#494) and will ship in the next release. I will follow up here when that build is out. On the network side you can stand your team down. The only caveat worth noting: a bare "EOF" means the SSH connection was torn down, so an oversized SSH packet crossing the inter-site link is not something the logs let us rule out completely. The trigger is squarely our oversized command, and the smaller commands also keep each SSH packet small, so this is resolved either way. Thanks again for the detailed diagnostics, they made this fast to pin down. |
Beta Was this translation helpful? Give feedback.
-
|
Good Day, We upgraded to version 1.4.5 and attempted the warm migration again on the same "large" VM. It unfortunately failed again. I tried it twice just to make sure. SSH is still enabled on the ESXi hosts. Here are the logs: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @neilcolyn-jpg, thanks for retesting on 1.4.5. Good news first: both earlier problems are gone. The disks zeroed cleanly (1.4.4 fix) and the run got past the "EOF" delta-apply step (1.4.5 fix), reaching the full copy and moving real data. What failed. The full copy of disk 1 (837 GB) hit a 12 hour per-operation safety timeout before finishing ("SSH connection timeout (43200s)" = 12h). It never completed, so the migration aborted there. Where the bottleneck is. Your log narrows it down: the zero step wrote the whole 837 GB to your iSCSI LVM in about 55 minutes (~250 MB/s), so the write path is fast. The full copy of that same disk then ran at under 19 MB/s, about 7x slower than your 1.4.4 run (~142 MB/s). The disk data never crosses the inter-site link (the node reads from vSphere and writes to the LVM, both in LD6; JH1 only holds the SSH control channel). So the slow part is the read from vSphere through VDDK on this attempt. Two quick things would settle it:
Steady-but-low write = progressing but read-starved (source/VDDK limit); little or no IO = stalled. On our side, we are adding live per-disk throughput to the log and replacing the flat 12 hour cap with a progress-based (inactivity) timeout, so a transfer that is genuinely moving is not cut off while a truly stalled one fails fast. Thanks again, the retests have been very helpful. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the quick response. There are no existing snapshots on the source VM on the VMware side. I reran the job and copied logs 2 minutes after the Full CBT copy began and 2 hours after the Full CBT copy which should now be replicating the large disk as we speak. However, it's probably going to run for a long time before failing... I'll keep running the commands every hour or so. I've attached the logs. Is there not a 12-hour limit on the replication job on the application side? SSH connection timeout (43200s). The live per-disk throughput metrics will be very useful. Kind Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Neil, thank you for the captures and for confirming there are no snapshots on the source. To answer your question directly: yes, the 12 hour limit is ours, and your math is correct. I checked the code: 43200s is a safety timeout we apply to the disk copy step (it exists so a copy cannot hold a vSphere snapshot open indefinitely). On the failed run, the disk 1 copy reached that limit and was stopped. I do not want to over-read two point-in-time captures, so I will not claim more than the data supports yet. What would settle it is seeing how this current run ends:
Once we have that, we will finalize the timeout change we mentioned (moving away from a flat cap) using your real numbers rather than an assumption. Thanks again for the thorough testing. Kind regards |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. I had a look the logs now again "According to AI" - its saying the data copy phase for all disks is completely finished. Here is how we know: No dd processes: There are zero active dd lines on your screen. Proxmox has completely finished pushing raw blocks into the target LVM storage. nbdkit is still idling: The core nbdkit process is still running because this is a warm migration. The tool intentionally keeps the connection to the source ESXi host open so that it can perform final, fast delta-syncs of any data that changed while the main copy was running. "Good news: On version 1.4.5, the bulk copy of Disk 1 (837 GB) completed successfully! It ran at a steady ~100 MB/s, taking exactly 3 hours and 4 minutes to finish (well under the 12-hour timeout limit). At 14:24, iostat throughput dropped to 0, %util fell to 0%, and the dd processes cleanly terminated. However, the migration wrapper hasn't advanced the UI or moved to the next phase—nbdkit is still idling in the background. Is the wrapper waiting on a specific event hook to finish the job?" 
Kind Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Neil, that capture is the one that pins it. Reading your numbers: the disk 1 copy did the work (around 100 MB/s) and the dd processes have exited, but the migration is not idle between phases, it is stuck. The single nbdkit still running is the disk 1 reader, and it is still up precisely because the step that would tear it down has not been reached yet. Here is what is happening. Each disk copy runs as one long SSH command from ProxCenter to the Proxmox node. A plain dd produces no output until it finishes, so for the whole multi-hour copy that SSH control connection carries no traffic. An idle connection that long is being torn down somewhere on the path (the JH1 to LD6 link is the likely place, but the trigger is on our side: our SSH client was not sending keepalives, so nothing kept the session warm and nothing detected the drop). The dd finishes on the node, but the "finished" signal has no live connection to come back on, so the migration waits, and the only thing that eventually unblocks it is the 12 hour safety cap, which is where your earlier "SSH connection timeout (43200s)" came from. Two practical points:
One thing that would help confirm the picture: could you ask your network team for the idle session timeout (the TCP idle timeout for an established session)? On 1.4.4 your large disk copied in roughly 100 minutes and succeeded; on 1.4.5 it ran about 3 hours and was dropped, so that timeout almost certainly sits between those two. If you want to retry before our build is out, keeping that SSH path from going idle would avoid the drop: a longer idle-session timeout for that flow, or running the migration with ProxCenter on the same site as the node. Thanks again. Kind regards |
Beta Was this translation helpful? Give feedback.
-
|
Good Day, Just a quick check-in to confirm, we still waiting on fix #528 to be released? Kind Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Neil, Yes, that's correct. The fix in #528 is merged but has not shipped yet. The latest published build is v1.4.5 (27 June), which predates the merge, so #528 is not in it. It is queued for the next release, v1.4.6, which we are targeting before the end of the month. That build bundles a number of bug fixes along with new features, including a High Availability mode for ProxCenter itself. I will follow up on this thread as soon as it is out. If you want to retry before then, the earlier workaround still stands: keep the SSH control path from going idle during the long copy, either by raising the idle-session timeout for that flow on the JH1 to LD6 link, or by running ProxCenter on the same site as the Proxmox node. That avoids the drop that #528 fixes properly. Kind regards |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Good Day,
We are getting an error when attempting a warm migration of a large VM.
Retried with same error.
Source Platform:
vSphere 8.0.3
Storage: 9 Disks with 1.43TB storage
Destination:
Proxmox 9.1.9
Destination Storage: iSCSI LVM thick (11TB Free space)
Error Log:
[09:16:03] Warm migration: planning
[09:16:05] ✓ Authenticated to 10.205.3.100 as proxcenter@vsphere.local
[09:16:08] ✓ VDDK preflight OK on Proxmox node
[09:16:14] ✓ CBT enabled on source
[09:16:15] ✓ Target VM 116 created on ld6-pve-node01
[09:18:54] Disk 0: zeroed thick target /dev/vg-ld6-isp-t1-net-ab31-is-mpatha/vm-116-disk-0
[09:18:54] Disk 0: target LD6-ISP-T1-NET-AB31-IS-LVM-TK:vm-116-disk-0 → /dev/vg-ld6-isp-t1-net-ab31-is-mpatha/vm-116-disk-0 (128.0 GB)
[10:04:05] ✗ Warm migration failed: Failed to zero thick target /dev/vg-ld6-isp-t1-net-ab31-is-mpatha/vm-116-disk-1 before warm copy (unwritten regions would expose stale data): Exit code 1
Beta Was this translation helpful? Give feedback.
All reactions