fix(update): fan-out downloads get full bytes; stuck updates explain themselves

.changeset/node-update-fanout-and-restart-guidance.md

@@ -0,0 +1,8 @@
+---
+"@internal/server": patch
+"@internal/node": patch
+---
+
+Fix "Update all" handing every node but the first an empty download. When several machines asked at once, the shared fetch handed out one spent stream; each caller now opens its own read of the finished download, and a digest mismatch fails the fetch before any byte is served.
+
+Say more when an update refuses or stalls. The "not supervised" refusal now names whether a service definition exists on that machine and, for a launchd domain that needs someone signed in at the display, says so. When the swap lands but the restart is refused, the node reports the new binary as installed, shows the manager's own reason, and names the two ways to get it running.

apps/node/agent/docs/service.md

@@ -152,6 +152,12 @@ stopped. Start is `bootstrap` falling back to
 `kickstart -k`; restart is `kickstart -k` (a bare `kickstart` on a running job
 changes nothing); stop is `bootout` (KeepAlive undoes a mere kill), and on
 Linux `stop`, never `disable --now`: un-enabling is what uninstall is for.
+A `kickstart` that answers `Domain does not support specified action` (exit
+125) is the GUI-session half of the domain rule: `gui/<uid>` exists only while
+someone is signed in at the display, so an SSH-only operator gets that refusal
+and a hint to run `subshell run` by hand (measured on a Mac Studio,
+2026-10-23); the hint ships in the error line because launchd's own words
+name no remedy.
 
 **The pane guard is the reason those live here rather than in the caller.**
 This node's subshells run their tmux servers as CHILDREN of the daemon, so a

apps/node/agent/docs/update.md

@@ -73,6 +73,19 @@ budget, and it is a belt: the plane pushes `set_allowed_dirs` on every accepted
 of the timer never firing is a stale ~70 MB `.previous`; the cost of it firing
 early is the rollback.
 
+**A restart that fails AFTER the swap is a report, not a refusal.** The
+keyboard path's `restart` is not an all-or-nothing gate: the swap is already
+done, and the manager's refusal to bounce is its own outcome. `applyUpdate`
+then returns `restarted: false` with a `note` that says all of it in order:
+the new binary IS installed, the running daemon is still the old one, the
+manager's own first line is the reason, and the two ways forward (`subshell
+service restart`, or stop and restart the running `subshell run`) are named.
+Bare-manager-line notes read as "the update failed", which on 2026-10-23 left
+a Mac Studio's operator standing next to an installed 2.3.1 trying to
+re-download it. The launchd domain cause (kickstart exit 125: a user's `gui/
+<uid>` agents need someone signed in at the display) gets its own sentence
+from `controlService`, which is where launchd's words are first-hand.
+
 **Refusals are the WIRE CONSTANTS, never sentences.** The plane matches
 `NodeRpcError.detail` by equality, so `applyUpdate` throws `UpdateRefused`
 carrying a `NODE_RESULT_*` string alongside the human message, and the executor

apps/node/agent/src/__tests__/service.test.ts

@@ -1194,6 +1194,83 @@ describe("controlService", () => {
     expect(s.calls.at(-1)).toEqual(["launchctl", "kickstart", "-k", TARGET]);
   });
 
+  // Live incident 2026-10-23 (thedarknesss-Mac-Studio): a manual update swapped
+  // the binary, then `kickstart -k` answered 125 "Domain does not support
+  // specified action" — the gui/<uid> agent domain exists only while someone is
+  // signed in at the machine's own display, so a daemon started from an SSH
+  // session sits outside launchd's reach. The bare exit code told nobody that,
+  // or what to do.
+  test("darwin: a kickstart DOMAIN failure names the GUI-session cause and the way through", async () => {
+    const s = stub({
+      platform: "darwin",
+      respond: (cmd) =>
+        cmd[0] === "plutil"
+          ? { code: 0, out: "true\n", err: "" }
+          : cmd[1] === "print"
+            ? { code: 0, out: "\tstate = running\n\tpid = 5150\n", err: "" }
+            : cmd[1] === "kickstart"
+              ? {
+                  code: 125,
+                  out: "",
+                  err: 'Could not kickstart service "dev.subshell.client": 125: Domain does not support specified action\n',
+                }
+              : { code: 0, out: "", err: "" },
+    });
+    s.files.set(PLIST, "<key>AbandonProcessGroup</key><true/><key>RunAtLoad</key><true/>");
+    const r = await controlService(s.deps, "restart");
+    expect(r.code).toBe(1);
+    expect(msgLine(r.err)).toContain("launchctl kickstart -k failed (exit 125)");
+    expect(msgLine(r.err)).toContain("Domain does not support specified action");
+    expect(msgLine(r.err)).toContain("signed in at this machine's own display");
+    expect(msgLine(r.err)).toContain("subshell run");
+  });
+
+  test("darwin: other kickstart failures point at service status, not the domain story", async () => {
+    const s = stub({
+      platform: "darwin",
+      respond: (cmd) =>
+        cmd[0] === "plutil"
+          ? { code: 0, out: "true\n", err: "" }
+          : cmd[1] === "print"
+            ? { code: 0, out: "\tstate = running\n\tpid = 5150\n", err: "" }
+            : cmd[1] === "kickstart"
+              ? { code: 1, out: "", err: "BootStrap failed for some other reason\n" }
+              : { code: 0, out: "", err: "" },
+    });
+    s.files.set(PLIST, "<key>AbandonProcessGroup</key><true/><key>RunAtLoad</key><true/>");
+    const r = await controlService(s.deps, "restart");
+    expect(r.code).toBe(1);
+    expect(msgLine(r.err)).toContain("BootStrap failed for some other reason");
+    expect(msgLine(r.err)).toContain("subshell service status");
+    expect(msgLine(r.err)).not.toContain("signed in at");
+  });
+
+  // The GUI story claims someone needs to sign in at the display, which is
+  // true only of the "Domain does not support specified action" refusal. A
+  // near-miss that merely mentions a domain must not borrow the story.
+  test("darwin: a domain-word failure that is not the GUI-domain refusal gets the status pointer", async () => {
+    const s = stub({
+      platform: "darwin",
+      respond: (cmd) =>
+        cmd[0] === "plutil"
+          ? { code: 0, out: "true\n", err: "" }
+          : cmd[1] === "print"
+            ? { code: 0, out: "\tstate = running\n\tpid = 5150\n", err: "" }
+            : cmd[1] === "kickstart"
+              ? {
+                  code: 3,
+                  out: "",
+                  err: 'Could not find service "dev.subshell.client" in domain for uid: 1000\n',
+                }
+              : { code: 0, out: "", err: "" },
+    });
+    s.files.set(PLIST, "<key>AbandonProcessGroup</key><true/><key>RunAtLoad</key><true/>");
+    const r = await controlService(s.deps, "restart");
+    expect(r.code).toBe(1);
+    expect(msgLine(r.err)).toContain("subshell service status");
+    expect(msgLine(r.err)).not.toContain("signed in at");
+  });
+
   test("darwin: restart of a STOPPED job bootstraps it — kickstart needs a loaded job", async () => {
     const s = darwinServiceStub({ loaded: false });
     const r = await controlService(s.deps, "restart");

apps/node/agent/src/__tests__/update.test.ts

@@ -670,7 +670,45 @@ describe("applyUpdate", () => {
       restartService: async () => ({ code: 1, out: "", err: "subshell: this would close 3 subshells\nuse --force" }),
     });
     expect(applied.restarted).toBe(false);
-    expect(applied.note).toBe("subshell: this would close 3 subshells");
+    // The note says ALL of it, in the order the person needs it: the swap
+    // landed, the running daemon is the OLD one, the manager's own words are
+    // the reason, and two ways forward are named. A bare manager line read
+    // like nothing had happened.
+    expect(applied.note).toContain("the new binary is installed");
+    expect(applied.note).toContain("this would close 3 subshells");
+    expect(applied.note).toContain("subshell service restart");
+    expect(await readFile(binary, "utf8")).toBe("NEXT");
+  });
+
+  it("the refused-restart note names the swap first — the Mac-Studio kickstart shape", async () => {
+    // Live incident 2026-10-23: `launchctl kickstart -k` answered 125 ("Domain
+    // does not support specified action") after a manual update had ALREADY
+    // replaced the binary. The operator read that as a failed update; the
+    // daemon was one restart away from the new version. The note must make
+    // the installed-and-standing-still state unmistakable.
+    const { binary, root, dataDir } = await installedNode();
+    pretendInstalledAt(binary);
+    const local = join(root, "next");
+    await writeFile(local, "NEXT");
+    const applied = await applyUpdate({
+      binaryDeps: NO_SERVICE_DEFINITION,
+      source: { kind: "file", path: local },
+      version: "0.9.3",
+      force: true,
+      restart: true,
+      origin: "cli",
+      dataDir,
+      probeVersion: async () => "0.9.3",
+      restartService: async () => ({
+        code: 1,
+        out: "",
+        err: 'subshell: launchctl kickstart -k failed (exit 125): Could not kickstart service "dev.subshell.client": 125: Domain does not support specified action',
+      }),
+    });
+    expect(applied.restarted).toBe(false);
+    expect(applied.note).toContain("the new binary is installed");
+    expect(applied.note).toContain("still runs");
+    expect(applied.note).toContain("launchctl kickstart -k failed (exit 125)");
     expect(await readFile(binary, "utf8")).toBe("NEXT");
   });
 });

apps/node/agent/src/service.ts

@@ -1314,7 +1314,23 @@ export async function controlService(
     if (state.state === "stopped")
       return bootstrapDarwin(deps, target, DONE.restart, warning, state.definitionPath ?? plistPath(deps.home));
     const res = await deps.runCmd(["launchctl", "kickstart", "-k", target]);
-    if (res.code !== 0) return errLine(`launchctl kickstart -k failed (exit ${res.code}): ${cmdDetail(res)}`);
+    if (res.code !== 0) {
+      // A bare launchctl exit code is not an action. "Domain does not support
+      // specified action" (125) is the common one on a headless Mac: the
+      // gui/<uid> agent domain exists only while someone is signed in at the
+      // machine's own display, so a daemon started over SSH sits outside
+      // launchd's reach and no `-k` will ever land. Only that exact sentence
+      // earns the GUI-session story; other launchctl failures that merely
+      // mention a domain (the "find service ... in domain" family) get the
+      // status pointer instead, because the sign-in story would be false for
+      // them. Everything else at least gets pointed at the status view,
+      // which carries launchd's own words.
+      const detail = cmdDetail(res);
+      const hint = /domain does not support specified action/i.test(detail)
+        ? "; the user's launchd domain needs someone signed in at this machine's own display (its agents run in a GUI session, not an SSH one). Start the daemon by hand with `subshell run`, or sign in at the console and try again"
+        : "; run `subshell service status` to see what launchd says about this job";
+      return errLine(`launchctl kickstart -k failed (exit ${res.code}): ${detail}${hint}`);
+    }
     return done(DONE.restart);
   }
   // start

apps/node/agent/src/update.ts

@@ -820,12 +820,21 @@ export async function applyUpdate(input: ApplyUpdateInput): Promise<AppliedUpdat
   }
   const res = await input.restartService(input.force === true);
   if (res.code !== 0) {
+    // The swap already happened, and the manager's line alone ("launchctl
+    // kickstart -k failed (exit 125): ...") leaves the machine's actual state
+    // a riddle: the FILE on disk is the new version, the RUNNING daemon is
+    // the old one, and that split is the fact the person needs first. The
+    // manager's own words ride along because the reason lives in them.
+    const managerLine =
+      (res.err.trim() || res.out.trim() || "the service manager would not restart it").split("\n")[0] ?? "";
     return {
       from: NODE_VERSION,
       to: input.version,
       binary,
       restarted: false,
-      note: (res.err.trim() || res.out.trim() || "the service manager would not restart it").split("\n")[0] ?? "",
+      note:
+        `the new binary is installed, but the daemon still runs ${NODE_VERSION}: ${managerLine}. ` +
+        "Restart it where it runs: `subshell service restart`, or stop and start the running `subshell run` again",
     };
   }
   return { from: NODE_VERSION, to: input.version, binary, restarted: true };

apps/server/api/AGENTS.md

@@ -121,8 +121,11 @@ The Nodes plane serves the prebuilt `subshell` binaries from
 `NODE_ARTIFACTS_DIR` at `GET /api/downloads/node/*` (cookie,
 unconsumed setup key, or a still-valid one-time `?update_token=`; never
 anonymous), and fetches the binary LAZILY from the project's own
-`cli-node-v*` GitHub release when that is the answer, streaming it through
-while hashing against the SIGNED manifest's `assets` map. Which release a
+`cli-node-v*` GitHub release when that is the answer, downloading it in full
+and hashing the whole file against the SIGNED manifest's `assets` map before
+any byte is served (the 2026-10-23 fan-out fix: an in-flight fetch is
+shared, and a shared one-shot stream handed every joiner but the first an
+empty body). Which release a
 node is OFFERED (`NODE_PROTOCOL_VERSION` matching), the three-way refusal
 grammar, the shelf-reconciled versus release-coherent serving (spec
 2026-10-09: the copy is asked what it is before it wins), and the surfaces

apps/server/api/docs/node-artifacts.md

@@ -16,13 +16,15 @@ still lazy: served on request, never warmed up, no admin button, no poll, so a
 plane whose nodes are all one platform never spends a byte on the others.
 Since 2026-10-09 the tiny index/manifest reads behind the serve decision run
 on EVERY authenticated download request (TTL-cached; a failed read is backed
-off a minute), not just on the branch that ends in a fetch. The bytes
-stream THROUGH while being hashed against the digest from the release's
-SIGNED manifest `assets` map; the manifest and its signature are
-verified before the first binary byte, and the `.sha256` sidecar is never
-fetched on this path; a mismatch errors the response mid-flight, so
-nothing unverified is cached and the node's own digest check before
-`chmod +x` still decides.
+off a minute), not just on the branch that ends in a fetch. The bytes are
+downloaded IN FULL and hashed against the digest from the release's
+SIGNED manifest `assets` map before anything is served; the manifest and its
+signature are verified before the first binary byte, and the `.sha256` sidecar
+is never fetched on this path. A mismatch never reaches a response at all: the
+fetch itself fails, the partial temp is deleted, and nothing unverified is
+cached (since the 2026-10-23 fan-out fix; the streaming-through shape shared
+one dead stream with every concurrent caller). The node's own digest check
+before `chmod +x` still decides.
 `SUBSHELL_RELEASE_URL` configures it and EMPTY disables it (the
 air-gapped configuration, and the default under `IS_TEST` so no suite reaches
 the network by accident). (It was `SUBSHELL_NODE_RELEASE_URL` until spec
@@ -62,11 +64,25 @@ instead** (2026-09-21, the live-incident follow-up). The token binds the digest
 the update command named from the signed manifest, so the disk file is served
 only when it hashes to it, and anything else is fetched around: an absent disk
 file is cached exactly as the installer lazy fetch always did, while a
-present-but-different file is streamed through and never overwritten (the
-hand-published-binary rule holds). A plane that cannot fetch refuses a token
+present-but-different file is served from a temp copy and never overwritten
+(the hand-published-binary rule holds; the temp goes when the response ends).
+A download killed outright strands its uniquely-named temp, because no
+in-process path cleans what a SIGKILL interrupted, so each download sweeps
+strays older than an hour first; the age bound is what makes that safe beside
+a concurrent fetch's live copy, whose mtime stays fresh while it is written.
+Since the 2026-10-23 fan-out fix the fetch lands COMPLETE and verified on disk
+before the route hands anything out: the release-vs-token digest comparison
+therefore runs with no byte served, a mismatch answers 409, and an
+update that ordered a release the source has moved past learns it before its
+download rather than after it. When the disk had nothing, the fetched release
+stays cached: it is what the next installer gets, and the retry fast-paths
+onto it. A plane that cannot fetch refuses a token
 download whose disk copy is stale with a 409 naming the remedy, and the
 narrowed guard in `update-node.route.ts` is that case's backstop at the update
-route itself.
+route itself. The coalescer shares the settled download, not a stream:
+concurrent same-target fetches (an "Update all" fan-out) each open their own
+read of it, and a `cache: false` fetch runs uncoalesced because its temp
+cleanup belongs to that one caller.
 
 Three surfaces still report what is on disk, so the air-gapped case is visible
 instead of being discovered: the published set (`lib/node-artifacts.ts:publishedNodeTargets`,

apps/server/api/docs/self-management.md

@@ -247,6 +247,16 @@ which silently asserted over the no-report case), and the server's OWN
 `restart`/`update` refusals follow the same rule on their own
 `paneSafety` field.
 
+The same honesty shapes the `NODE_NOT_SUPERVISED` sentence on the UPDATE
+route (`update-node.route.ts`; the service verb keeps its plain one): it
+composes from the runtime report already in hand, so
+"installed but not running from it" and "nothing installed" name different
+remedies, a launchd domain that answers `unknown` adds its own clause (its
+user agents need someone signed in at the display), and no facts at all keep
+the plain sentence. The report is read live-or-held: a refused agent's `ready`
+is that machine's last honest answer, persisted before the gate that held it,
+and a held node is exactly the machine someone presses Update on.
+
 `runtime` (the node's report of how its own process runs) lives on the LIVE
 CONNECTION, never in the `nodes` table, and `GET /api/nodes/:id` exposes it
 only when the node is online, the viewer can configure it, and the row is an