TunnelManager agent doesn't auto-recover a dead-but-still-present tunnel (reboot test only covers explicit refresh)

[danielpaulus]

Summary

The running tunnel agent (ios tunnel start) never automatically rebuilds a tunnel whose underlying transport has died while the device is still present on USB — e.g. after a quick device reboot that usbmux doesn't observe as a disconnect, or a tunnel transport death with no USB drop. The agent keeps serving the stale RSD record forever, and every consumer dialing it fails (see #764 for the 135s-no-timeout amplification).

TestTunnelRebootRecovery exists but does not guard this path: it recovers by explicitly calling ios tunnel refresh, which the production agent never does. So the on-demand command is tested; the agent's automatic self-heal is not.

Live evidence (real CI host)

On the Linux tunnel runner, device 00008110-001C19A43420401E (iPhone14,2, iOS 18.4.1):

• usbmux lists the device fine — it never left the device list.
• The agent still serves the stale record {"address":"fdb2:7531:10d5::1","rsdPort":51971,"userspaceTun":false} — the exact dead endpoint e2e tests timed out on across two separate runs.
• Because the record persisted unchanged, the agent's cleanup (which only stops tunnels for devices that vanish from usbmux) never fired, and UpdateTunnels skips any device that already has a record (if _, exists := localTunnels[udid]; exists { continue }) — so it was never rebuilt.

Root cause

TunnelManager.UpdateTunnels:

• creates a tunnel only for a device with no existing record, and
• tears down a record only when the device disappears from ListDevices().

There is no liveness/health check of an existing tunnel record. A tunnel that dies without a corresponding usbmux disconnect becomes a permanent stale entry. #738 added reboot recovery, but only for the disconnect-driven path (device leaves USB → cleanup → rebuild) and the on-demand tunnel refresh command.

TestTunnelRebootRecovery even acknowledges the gap and works around it:

} else {
    t.Logf("did not observe %s leave USB (it may have rebooted quickly); proceeding to recovery", udid)
}
// ... then explicitly:
harness.TryRun(t, "tunnel", "refresh", "--udid="+udid, portArg)

Proposed fix

1. Agent-side auto health-check + refresh. In the UpdateTunnels loop, periodically probe existing tunnel records (cheap RSD/RemoteXPC handshake or a short-timeout dial). On failure, treat the record as stale → tear it down and rebuild for that device (reuse Add per-device tunnel stop and refresh #738's per-device refresh). This makes the agent do automatically what the test does by hand.
2. Pair with the fast-fail dial timeout from Kernel tunnel: device-unreachable mid-run causes 135s no-timeout RSD dials → e2e job timeout #764 so a dead tunnel is detected in seconds, not the ~135s kernel default.
3. Strengthen the test. Extend TestTunnelRebootRecovery (or add a sibling) to assert the agent recovers a rebooted device's tunnel without an explicit tunnel refresh call — i.e. the running agent self-heals on its own. Today the test passing gives false confidence.

Related

• Kernel tunnel: device-unreachable mid-run causes 135s no-timeout RSD dials → e2e job timeout #764 — kernel-tunnel dead-endpoint dials block ~135s (no dial timeout); same incident, complementary fix (fast-fail + staleness signal).
• Add per-device tunnel stop and refresh #738 — added per-device tunnel stop/refresh and the disconnect-driven recovery this issue extends.