Skip to content

Commit 14e26bc

Browse files
authored
Merge pull request #27 from Demali-876/feat/node-service-verify
feat(node): service command that proves headless boot
2 parents fffa264 + dd39fcc commit 14e26bc

12 files changed

Lines changed: 439 additions & 8 deletions

File tree

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ Hosted by `runtime/server.ts` (Fastify + `@fastify/websocket`) and the same eval
9393
In production:
9494
- `ecosystem.config.cjs` configures PM2 to run `<install-dir>/current/src/supervise.ts`, which runs the control tunnel (`bun run control`, the data path) and a loopback-only runtime server (`bun run start`) as one unit and exits if either does, so an `update_apply` (or a crash) restarts both from the refreshed `current`. The client-facing data plane is bridged onto the control tunnel by the orchestrator node-gateway, so the node opens no inbound port and terminates no TLS. The `systemd/` unit execs the same entry point via its `#!/usr/bin/env bun` shebang, and the macOS LaunchDaemon runs `pm2-runtime` against this same config.
9595
- `scripts/install-release.sh` is the default installer: unpacks the verified tarball into `releases/<version>/`, installs prod deps with the lockfile, atomically moves the `current` symlink, then prunes old releases per `CONSENSUS_NODE_RELEASE_RETENTION` (default 3) — while protecting the release that is mid-update.
96-
- `scripts/ensure-pm2.sh` and `scripts/start-pm2.sh` bootstrap PM2 on macOS (Homebrew → Node → PM2). For boot persistence WITHOUT a login, `scripts/install-launchd.sh` (macOS, needs sudo) renders `launchd/com.consensus.node.plist.template` into `/Library/LaunchDaemons` and runs `pm2-runtime` under it; on Linux use `systemd/consensus-node.service`. Do NOT use `pm2 startup` on macOS — it emits a LaunchAgent, which loads only at user login. The installer runs `bun run secrets:check` **as the daemon's account** first and refuses to install if the encryption data key is not readable without a login. **FileVault must be off on a node**: it halts at a pre-boot unlock prompt, so nothing — daemon or agent — runs until a human types the password.
96+
- `scripts/ensure-pm2.sh` and `scripts/start-pm2.sh` bootstrap PM2 on macOS (Homebrew → Node → PM2). For boot persistence WITHOUT a login, `scripts/install-launchd.sh` (macOS, needs sudo) renders `launchd/com.consensus.node.plist.template` into `/Library/LaunchDaemons` and runs `pm2-runtime` under it; on Linux use `systemd/consensus-node.service`. Do NOT use `pm2 startup` on macOS — it emits a LaunchAgent, which loads only at user login. The installer runs `bun run secrets:check` **as the daemon's account** first and refuses to install if the encryption data key is not readable without a login. **FileVault must be off on a node**: it halts at a pre-boot unlock prompt, so nothing — daemon or agent — runs until a human types the password. `scripts/node-service.sh` operates the unit (`restart`/`status`/`start`/`stop`/`ping`/`logs`); `restart` waits for the orchestrator to report the node active again, which is the only signal that it actually came back rather than merely relaunching.
9797

9898
The wrapper still tolerates the legacy exit code `75` from older releases. New code should exit with `0` (the supervisor handles the restart) and close with WS code `1012` so the server distinguishes update shutdowns from crashes.
9999

README.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -233,3 +233,26 @@ Two prerequisites for a truly headless node:
233233
`sudo fdesetup authrestart` boots once unattended, but power loss still needs a human.
234234
- **Automatic restart after a power cut**, so the machine comes back at all:
235235
`sudo pmset -a autorestart 1` (and `sudo pmset -a sleep 0` to stop it sleeping).
236+
237+
### Operating and proving it
238+
239+
```bash
240+
sudo scripts/node-service.sh restart # relaunch, then wait for the node to serve
241+
sudo scripts/node-service.sh status
242+
scripts/node-service.sh ping # is the orchestrator seeing this node?
243+
scripts/node-service.sh logs
244+
```
245+
246+
`restart` relaunches the unit and then waits for the **orchestrator** to report this
247+
node `active` again. That distinction matters: a live pid only says the unit
248+
relaunched, while the orchestrator confirms the node reconnected and is serving.
249+
`ping` asks the same question on its own.
250+
251+
There is deliberately no command that claims to prove the node started *before* a
252+
login. Whether a given start was pre- or post-login is not something the machine can
253+
report reliably after the fact, so the check that earns its keep is operational —
254+
restart it and confirm it comes back and serves.
255+
256+
Existing nodes that predate the boot unit do not need the wizard: the supervisor logs
257+
`headless-boot-NOT-configured` at startup and `/health` reports `headless_boot`, so an
258+
operator can see which nodes would not survive an unattended reboot.

scripts/install-launchd.sh

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,16 @@ echo " bun: ${bun_bin}"
134134
echo " install dir: ${install_dir}"
135135
echo " state dir: ${state_dir}"
136136
echo
137-
echo "Status: sudo launchctl print system/${LABEL}"
138-
echo "Logs: tail -f ${state_dir}/launchd.err.log"
137+
echo "Status: sudo scripts/node-service.sh status"
138+
echo "Restart: sudo scripts/node-service.sh restart"
139+
echo "Ping: scripts/node-service.sh ping"
140+
echo "Logs: scripts/node-service.sh logs"
139141
echo
140-
echo "It now starts at boot with no login required. Verify with a reboot."
142+
echo "This is a LaunchDaemon, so it starts at boot without a login (unlike"
143+
echo "\`pm2 startup\`, which writes a login-gated LaunchAgent on macOS)."
144+
echo
145+
echo "Check it end to end now:"
146+
echo " sudo scripts/node-service.sh restart"
147+
echo
148+
echo "That relaunches the unit and waits for the orchestrator to report this node"
149+
echo "active again — the node is not really back until the orchestrator sees it."

scripts/node-service.sh

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Operate the Consensus node boot service on macOS.
4+
#
5+
# sudo scripts/node-service.sh restart # restart, then wait for the node to serve
6+
# sudo scripts/node-service.sh status
7+
# sudo scripts/node-service.sh start|stop
8+
# scripts/node-service.sh ping # ask the orchestrator if this node is live
9+
# scripts/node-service.sh logs
10+
#
11+
# `restart` is the useful one: it relaunches the unit and then waits for the
12+
# ORCHESTRATOR to report this node active again. A live pid only says the unit
13+
# relaunched; only the orchestrator confirms the node reconnected and is serving.
14+
#
15+
# There is deliberately no "prove it started before login" command. Whether a given
16+
# start happened pre- or post-login is not something this machine can report
17+
# reliably after the fact, so the test that earns its keep is operational: restart
18+
# it and confirm the node comes back and serves.
19+
#
20+
# On Linux, use systemctl against systemd/consensus-node.service instead.
21+
#
22+
set -uo pipefail
23+
24+
LABEL="com.consensus.node"
25+
PLIST="/Library/LaunchDaemons/${LABEL}.plist"
26+
SERVICE="system/${LABEL}"
27+
28+
ONLINE_TIMEOUT_SECONDS="${CONSENSUS_ONLINE_TIMEOUT_SECONDS:-90}"
29+
DEFAULT_SERVER_URL="https://consensus.canister.software"
30+
31+
red() { printf '\033[31m%s\033[0m\n' "$*"; }
32+
green() { printf '\033[32m%s\033[0m\n' "$*"; }
33+
warn() { printf '\033[33m%s\033[0m\n' "$*"; }
34+
35+
need_root() {
36+
if [[ "${EUID}" -ne 0 ]]; then
37+
echo "This needs root (it talks to launchd's system domain):" >&2
38+
echo " sudo $0 $1" >&2
39+
exit 77
40+
fi
41+
}
42+
43+
service_pid() {
44+
launchctl print "${SERVICE}" 2>/dev/null | sed -n 's/^[[:space:]]*pid = \([0-9]*\).*/\1/p' | head -1
45+
}
46+
47+
state_dir_from_plist() {
48+
[[ -f "${PLIST}" ]] || return 1
49+
/usr/libexec/PlistBuddy -c "Print :EnvironmentVariables:CONSENSUS_STATE_DIR" "${PLIST}" 2>/dev/null
50+
}
51+
52+
server_url_from_plist() {
53+
[[ -f "${PLIST}" ]] || { echo "${DEFAULT_SERVER_URL}"; return; }
54+
/usr/libexec/PlistBuddy -c "Print :EnvironmentVariables:CONSENSUS_SERVER_URL" "${PLIST}" 2>/dev/null \
55+
|| echo "${DEFAULT_SERVER_URL}"
56+
}
57+
58+
node_id_from_state() {
59+
local state_dir; state_dir="$(state_dir_from_plist)" || state_dir="${HOME}/.consensus/node"
60+
[[ -f "${state_dir}/config.json" ]] || return 1
61+
/usr/bin/python3 -c "import json,sys; print(json.load(open(sys.argv[1])).get('node_id') or '')" \
62+
"${state_dir}/config.json" 2>/dev/null
63+
}
64+
65+
# The signal that matters: does the orchestrator consider this node live?
66+
orchestrator_status() {
67+
local node_id server
68+
node_id="$(node_id_from_state)" || return 1
69+
[[ -n "${node_id}" ]] || return 1
70+
server="$(server_url_from_plist)"
71+
curl -fsS --max-time 10 "${server}/node/status/${node_id}" 2>/dev/null \
72+
| /usr/bin/python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('status','unknown'))" 2>/dev/null
73+
}
74+
75+
wait_until_online() {
76+
local deadline=$(( $(date +%s) + ONLINE_TIMEOUT_SECONDS )) status=""
77+
echo "Waiting for the orchestrator to report this node active (up to ${ONLINE_TIMEOUT_SECONDS}s)..."
78+
while (( $(date +%s) < deadline )); do
79+
status="$(orchestrator_status)" || status=""
80+
if [[ "${status}" == "active" ]]; then
81+
green " orchestrator reports: active — the node reconnected and is serving"
82+
return 0
83+
fi
84+
sleep 5
85+
done
86+
warn " orchestrator did not report 'active' within ${ONLINE_TIMEOUT_SECONDS}s (last: ${status:-unreachable})"
87+
echo " Check the logs: scripts/node-service.sh logs"
88+
return 1
89+
}
90+
91+
cmd_status() {
92+
need_root status
93+
if [[ ! -f "${PLIST}" ]]; then
94+
red "not installed — ${PLIST} is missing"
95+
echo "This node will NOT come back after a reboot until someone logs in."
96+
echo "Install the boot service with: sudo scripts/install-launchd.sh"
97+
return 1
98+
fi
99+
echo "plist: ${PLIST}"
100+
local pid; pid="$(service_pid)"
101+
if [[ -z "${pid}" || "${pid}" == "0" ]]; then
102+
red "state: NOT RUNNING"
103+
launchctl print "${SERVICE}" 2>/dev/null | grep -E "last exit|state = " | sed 's/^[[:space:]]*/ /'
104+
return 1
105+
fi
106+
green "state: running (pid ${pid})"
107+
local started; started="$(ps -o lstart= -p "${pid}" 2>/dev/null | sed 's/ */ /g;s/^ *//;s/ *$//')"
108+
[[ -n "${started}" ]] && echo "since: ${started}"
109+
local remote; remote="$(orchestrator_status)" || remote=""
110+
if [[ "${remote}" == "active" ]]; then
111+
green "node: active on the orchestrator"
112+
elif [[ -n "${remote}" ]]; then
113+
warn "node: orchestrator reports '${remote}'"
114+
else
115+
warn "node: could not reach the orchestrator (or no node_id in state)"
116+
fi
117+
}
118+
119+
cmd_restart() {
120+
need_root restart
121+
[[ -f "${PLIST}" ]] || { red "not installed — run sudo scripts/install-launchd.sh"; exit 1; }
122+
echo "Restarting ${SERVICE}..."
123+
# kickstart -k kills the running instance and starts a fresh one — the same thing
124+
# launchd does on a crash.
125+
launchctl kickstart -k "${SERVICE}" || { red "kickstart failed"; exit 1; }
126+
sleep 2
127+
local pid; pid="$(service_pid)"
128+
if [[ -z "${pid}" || "${pid}" == "0" ]]; then
129+
red "unit did not come back up"
130+
exit 1
131+
fi
132+
green "unit relaunched (pid ${pid})"
133+
echo
134+
wait_until_online
135+
}
136+
137+
cmd_start() { need_root start; launchctl bootstrap system "${PLIST}" && launchctl enable "${SERVICE}" && cmd_status; }
138+
cmd_stop() { need_root stop; launchctl bootout "${SERVICE}" && echo "stopped"; }
139+
140+
cmd_ping() {
141+
local status; status="$(orchestrator_status)" || status=""
142+
if [[ "${status}" == "active" ]]; then
143+
green "active — the orchestrator sees this node"
144+
return 0
145+
fi
146+
if [[ -n "${status}" ]]; then
147+
warn "orchestrator reports '${status}'"
148+
return 1
149+
fi
150+
red "could not reach the orchestrator, or no node_id in state"
151+
return 1
152+
}
153+
154+
cmd_logs() {
155+
local state_dir; state_dir="$(state_dir_from_plist)" || state_dir="${HOME}/.consensus/node"
156+
echo "following ${state_dir} logs (ctrl-c to stop)"
157+
tail -f "${state_dir}/launchd.err.log" "${state_dir}/launchd.out.log"
158+
}
159+
160+
case "${1:-}" in
161+
status) cmd_status ;;
162+
restart) cmd_restart ;;
163+
start) cmd_start ;;
164+
stop) cmd_stop ;;
165+
ping) cmd_ping ;;
166+
logs) cmd_logs ;;
167+
*)
168+
echo "Usage: $0 <restart|status|start|stop|ping|logs>" >&2
169+
echo >&2
170+
echo " restart relaunch the unit, then wait for the orchestrator to see it active" >&2
171+
echo " ping ask the orchestrator whether this node is live" >&2
172+
exit 64
173+
;;
174+
esac

src/node/headless.ts

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
// Whether this machine is configured to bring the node back after a reboot with no
2+
// login.
3+
//
4+
// This exists for nodes that were onboarded BEFORE the boot unit did — they will
5+
// never see the setup wizard again, so the runtime itself has to tell them. The
6+
// supervisor logs it at startup and /health reports it, which also lets an operator
7+
// (or the orchestrator) see which nodes would not survive an unattended reboot.
8+
//
9+
// Detection is by the presence of the boot unit the installers write. That is a
10+
// weaker signal than observing an actual boot — it says "configured to start at
11+
// boot", not "observed to have started at boot". Proving the latter needs
12+
// scripts/node-service.sh verify on a machine where nobody has logged in, which is
13+
// a question a running process cannot answer about itself.
14+
15+
import fs from "node:fs";
16+
17+
const LAUNCHD_UNIT = "/Library/LaunchDaemons/com.consensus.node.plist";
18+
const SYSTEMD_UNITS = [
19+
"/etc/systemd/system/consensus-node.service",
20+
"/lib/systemd/system/consensus-node.service",
21+
"/usr/lib/systemd/system/consensus-node.service",
22+
];
23+
24+
export interface HeadlessBootStatus {
25+
configured: boolean;
26+
mechanism: "launchd" | "systemd" | "unknown";
27+
unit: string | null;
28+
detail: string;
29+
}
30+
31+
function exists(file: string): boolean {
32+
try {
33+
return fs.existsSync(file);
34+
} catch {
35+
return false;
36+
}
37+
}
38+
39+
export function headlessBootStatus(): HeadlessBootStatus {
40+
if (process.platform === "darwin") {
41+
if (exists(LAUNCHD_UNIT)) {
42+
return {
43+
configured: true,
44+
mechanism: "launchd",
45+
unit: LAUNCHD_UNIT,
46+
detail: "LaunchDaemon installed; starts at boot without a login.",
47+
};
48+
}
49+
return {
50+
configured: false,
51+
mechanism: "launchd",
52+
unit: null,
53+
// Named explicitly because `pm2 startup` is the thing operators reach for, and
54+
// on macOS it writes a LaunchAgent, which loads only at user login.
55+
detail:
56+
"No LaunchDaemon. This node will NOT come back after a reboot until someone logs in. " +
57+
"Run: sudo scripts/install-launchd.sh (pm2 startup does NOT work here — it writes a login-gated LaunchAgent).",
58+
};
59+
}
60+
61+
if (process.platform === "linux") {
62+
const unit = SYSTEMD_UNITS.find(exists);
63+
if (unit) {
64+
return { configured: true, mechanism: "systemd", unit, detail: "systemd unit installed." };
65+
}
66+
return {
67+
configured: false,
68+
mechanism: "systemd",
69+
unit: null,
70+
detail:
71+
"No systemd unit. This node will NOT restart after a reboot. " +
72+
"Install systemd/consensus-node.service and enable it.",
73+
};
74+
}
75+
76+
return {
77+
configured: false,
78+
mechanism: "unknown",
79+
unit: null,
80+
detail: `Unsupported platform for headless boot: ${process.platform}`,
81+
};
82+
}

src/runtime/server.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import websocket from "@fastify/websocket";
33
import { loadConfig } from "../node/state";
44
import { releaseManifest } from "../node/manifest";
55
import { integrityPayload } from "../node/integrity";
6+
import { headlessBootStatus } from "../node/headless";
67
import { capabilitiesRecord } from "./capabilities";
78
import { registerBenchmarkRoutes } from "./benchmarks/routes";
89
import { registerProxyRoutes } from "./proxy-worker";
@@ -18,6 +19,7 @@ export async function buildServer() {
1819
const config = await loadConfig();
1920
return {
2021
status: "healthy",
22+
headless_boot: headlessBootStatus(),
2123
registered: Boolean(config.node_id),
2224
node_id: config.node_id ?? null,
2325
domain: config.domain ?? null,

0 commit comments

Comments
 (0)