fix(run): wait for guest DHCP lease before running --network commands - #41
Merged
Conversation
init-stage2 backgrounds the DHCP lease so the vsock agent answers ~0.5-2s sooner (dew-agent rides vsock, not eth0). The trade-off is that `dew run` delivers its command over vsock and executes it the moment the agent is reachable — which can be before the guest has an IP/route. A command that touches the network right away (apk add, npm install, curl) then fails with "Network unreachable" / "bad address" through no fault of the caller's. Verified on macOS 26.3.1 (build 25D771280a): the VZ NAT itself works, but `dew run --network` at T0 reliably had no IP and DNS failed (3/3 runs); `apk add` failed. The symptom is a boot-time race, not the macOS 26 NAT regression it is easily mistaken for. Close the race host-side: when the caller asked for network, block on the guest's lease before running their command. Reuse the /run/dew-net-pending marker init-stage2 already sets/clears and dew-oci-run already gates container launches on. Best-effort and bounded (~30s, matching dew-oci-run): a lease that never lands warns and proceeds rather than hanging, and the default networkless run skips the barrier entirely and stays fast. Tests: guestNetReadyCmd shape + bound + exit contract, netLeasePending warn/proceed decision, and a drift guard tying guestNetPendingMarker to the build.sh set/clear/poll sites so a marker rename fails a test in the same change. Verified e2e: with the barrier, apk add / DNS at T0 succeed 3/3; without it (dew 0.9.0) they fail 3/3.
The banner fired on every macOS 26 run and read as "your outbound is broken," which led users to blame the VZ NAT regression for what was actually the boot-time DHCP-lease race (now fixed by the run lease barrier). On 26.3.1 and 26.5.1 the NAT works fine. Reword to: note current builds are fine, state that dew now waits for the lease so a fast failure at boot is usually not the NAT, and only then point at Code-Hex/vz#218 for a failure that survives the barrier. No test binds the banner text, so this is comment/string-only.
There was a problem hiding this comment.
Pull request overview
This PR fixes a boot-time race where dew run --network <cmd> could start executing the user’s command as soon as the vsock agent is reachable, which can be before the guest DHCP lease (and route/DNS) is established. It adds a host-side “network lease barrier” that waits (bounded) for the guest’s /run/dew-net-pending marker to clear before running network-dependent commands, and it refines the macOS 26 NAT regression banner to reduce misattribution.
Changes:
- Add a bounded host-side wait in
cmd/dewfor the guest DHCP lease marker before executing--networkcommands. - Add darwin-only unit tests that lock in the guest marker contract and the barrier’s timeout/exit-code behavior.
- Reword the macOS 26 NAT warning banner to clarify the common boot-time race vs. true NAT regressions.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| internal/vm/darwin/darwin.go | Adjusts macOS 26 NAT warning messaging to reduce confusion with the boot-time DHCP race. |
| cmd/dew/main.go | Implements the --network DHCP-lease barrier (marker wait) before executing user commands. |
| cmd/dew/run_network_wait_test.go | Adds contract tests for the marker name and the barrier’s bounded wait + decision logic. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+183
to
+184
| fmt.Fprintln(os.Stderr, | ||
| " ⚠ macOS 26 had an Apple VZ NAT regression; guest outbound may be unreliable on some 26.x builds.") | ||
| " ⚠ some early macOS 26 builds had an Apple VZ NAT regression (guest gets a 192.168.64.x address but outbound times out). Current builds are fine.") |
Comment on lines
+1554
to
+1556
| func netLeasePending(res *RunResult, err error) bool { | ||
| return err == nil && res != nil && res.ExitCode != 0 | ||
| } |
Comment on lines
+1965
to
+1967
| if tokenSent && cfg.Network { | ||
| waitGuestNetwork(d, cfg.VsockPort, token, budget.window(netReadyWait)) | ||
| } |
- Skip the barrier when the --timeout budget is already expired. budget.window can return a non-positive duration past the deadline, which leaves the exec's TimeoutMs unset and lets the barrier run for the agent default (~30s) beyond an expired budget. Gating on !budget.expired() also guarantees a positive window. The foreground exec already returns timeoutErr in that state. - Soften the macOS 26 banner: "Current builds are fine" over-claimed (only 26.3.1/26.5.1 were verified) and contradicted the "deliberately hedged" note in the same block. Say "later builds repaired it" instead. - Clarify netLeasePending: it intentionally treats ANY clean non-zero exit as "could not confirm the lease landed", not just the cap-hit exit 1 — an agent-timeout kill returns a different code but still means the lease never landed, so warning is correct. Documented the deliberate breadth and pinned it with an exit-127 test case.
Comment on lines
+1570
to
+1581
| func waitGuestNetwork(d vm.VM, port uint32, token string, window time.Duration) { | ||
| conn, err := connectVsock(d, port) | ||
| if err != nil { | ||
| return | ||
| } | ||
| defer conn.Close() | ||
| ec, ea := argvOrShellWrap([]string{guestNetReadyCmd(netReadyDeciseconds)}) | ||
| res, err := execVsockConnArgv(conn, token, ec, ea, window) | ||
| if netLeasePending(res, err) { | ||
| fmt.Fprintln(os.Stderr, "dew: guest network lease still pending; running command anyway (outbound may fail)") | ||
| } | ||
| } |
…udget Second-round Copilot review on waitGuestNetwork: - The barrier's vsock connect used connectVsock's fixed 5s deadline, not the per-run window, so a dead agent could stall it ~5s past an exhausted --timeout. Use connectVsockDeadline(window) so the connect is bounded too. - budget.window can return a sub-millisecond (or non-positive) duration near the deadline; int(window/time.Millisecond) then truncates to 0, req.TimeoutMs is omitempty, and the barrier runs for the agent default (~30s) past budget. Add runBudget.netReadyWindow(): run the barrier only when the full window fits (no --timeout) or ≥ netReadyMinBudget (1s) remains, so the window handed down is always ≥ 1s (TimeoutMs ≥ 1000) and bounds the connect. Subsumes the previous !expired() guard. Tests: TestRunBudget_NetReadyWindow covers no-timeout / ample / sub-floor / expired. Verified e2e: barrier still fixes apk@T0; --timeout still bounds the run.
Comment on lines
+1596
to
+1607
| func waitGuestNetwork(d vm.VM, port uint32, token string, window time.Duration) { | ||
| conn, err := connectVsockDeadline(d, port, window) | ||
| if err != nil { | ||
| return | ||
| } | ||
| defer conn.Close() | ||
| ec, ea := argvOrShellWrap([]string{guestNetReadyCmd(netReadyDeciseconds)}) | ||
| res, err := execVsockConnArgv(conn, token, ec, ea, window) | ||
| if netLeasePending(res, err) { | ||
| fmt.Fprintln(os.Stderr, "dew: guest network lease still pending; running command anyway (outbound may fail)") | ||
| } | ||
| } |
Third-round Copilot review: waitGuestNetwork was not actually bounded to the caller's window. The connect could take up to window, then execVsockConnArgv could take another window plus hostReadGrace (~15s) via hostReadBudget — and worse, WriteJSON has no timeout at all, so an agent that accepts the connect then never drains the write would hang the run indefinitely. Bound the whole barrier — connect, request write, response read — by one absolute deadline (now+window): connectVsockDeadline caps the connect, and a conn.SetDeadline caps the exec I/O on the same conn, so their sum never exceeds window. netReadyWindow already guarantees window ≥ 1s, and the no---timeout window (35s) still sits above the guest loop's 30s self-terminate, so the common "lease pending" case still returns exit 1 and warns; the deadline only fires on a real stall. SetDeadline is best-effort (the vz vsock conn supports it; a conn that doesn't falls back to the internal budget). Test: TestWaitGuestNetwork_HardBoundedByWindow drives a stalled agent (a never-read net.Pipe peer) and asserts the barrier returns at ~window instead of hanging — it would block forever on the untimed write without the deadline.
Comment on lines
+1547
to
+1551
| // netReadyWindow returns the barrier's connect/exec window and whether to run it | ||
| // at all. It runs when the full netReadyWait fits (no --timeout) or at least | ||
| // netReadyMinBudget remains; a nearly- or already-exhausted budget (window below | ||
| // the floor, including the negative window window() yields past the deadline) | ||
| // skips the barrier. Subsumes the plain !expired() guard: whenever it says run, |
Comment on lines
+1599
to
+1602
| // past its deadline. netReadyWindow guarantees window ≥ netReadyMinBudget, so | ||
| // the guest's own ~30s loop still self-terminates first in the common | ||
| // no---timeout case (window 35s > 30s) — the conn deadline only fires on a real | ||
| // stall. SetDeadline is best-effort (ignored error): a conn that can't take one |
Fourth-round Copilot review, comment-only: - "negative window window() yields" → "negative value window() yields" - "no---timeout case" → "case without --timeout" No code change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
dew run --network <cmd>could start the user's command before the guest had an IP/route. init-stage2 backgrounds the DHCP lease so the vsock agent answers ~0.5-2s sooner (dew-agent rides vsock, not eth0), butdew rundelivers its command over vsock and executes it the moment the agent is reachable — which can be before the NIC is up. Commands that touch the network right away (apk add,npm install,curl) then fail with "Network unreachable" / "bad address".This is easily mistaken for the macOS 26 Apple VZ NAT regression — but it's a boot-time race, not a broken NAT.
Verified on macOS 26.3.1 (build 25D771280a)
The VZ NAT itself works fine (ping/DNS/TLS to real hosts all succeed once the lease lands). But at T0:
dew run --networkimmediate net probeapk addat T0Default networkless
dew runis unaffected and stays fast (~1.4s).Fix
Close the race host-side: when the caller asked for network, block on the guest's lease before running their command. Reuse the
/run/dew-net-pendingmarker init-stage2 already sets/clears anddew-oci-runalready gates container launches on. Best-effort and bounded (~30s, matching dew-oci-run) — a lease that never lands warns and proceeds rather than hanging.Second commit rewords the macOS 26 NAT banner: it fired on every run and read as "your outbound is broken," fueling the misattribution. Now it notes current builds are fine and that dew waits for the lease, so a fast failure at boot is usually not the NAT.
Tests
guestNetReadyCmdshape + bound + ready/timeout exit contractnetLeasePendingwarn/proceed decision tableguestNetPendingMarkerto thebuild.shset/clear/poll sites so a marker rename fails a test in the same changego vetgreenapk add jqat T0 works)