Machine-wide delivery service: clients register a delivery endpoint instead of N unsupervised per-client daemons

[XertroV]

The requirement

Mail must wake the agent — in all cases, without the agent needing to check its inbox.

That disqualifies poll_inbox / c2c poll-inbox as a delivery guarantee for any client: they require the model to choose to look. A mechanism only counts as a wake if an external event pushes the message into the agent's attention.

The problem with how we do it today

Every wake path is a separate, per-client, mostly unsupervised process, and each one is its own single point of failure:

Client	Wake mechanism	What must be alive
Kimi	REST prompt-injection (POST /api/v1/sessions/{id}/prompts)	a per-alias C2c_kimi_notifier fork+setsid daemon — unsupervised
Codex	app-server thread/inject_items + gated auto-turn	the managed deliver loop / app-server bridge
agy	agentapi send-message	the deliver-watch sidecar
OpenCode	deliver-watch sidecar	that sidecar
Claude	PostToolUse/Stop/SessionStart hooks	(fires only on agent activity)
Grok	hooks + c2c monitor	a monitor the agent had to arm

Consequences we have actually hit:

• kimi notifier strands pre-startup inbox backlog (silent for senders) #9 — a Kimi session was registered but its notifier was never armed; it stayed permanently DEAF while senders kept queuing mail. Nothing re-armed it because arming is one-shot with no supervision.
• Codex sessions silently lose delivery when hook config drifts after launch #27 — a managed Codex session went silently deaf when its deliver loop died while the frontend survived; the false-healthy classification hid it.
• Inverse leak — notifier daemons outlive their sessions (one observed polling since Jul 18) because teardown is per-alias and best-effort.
• N clients × per-alias daemons means N× the things that can die, each needing its own supervision, teardown, staleness detection, and doctor check.

Proposal — a machine-wide delivery service

One supervised, machine-wide singleton that owns delivery for every local session, with per-client endpoint adapters. Clients (or their managed launchers) register their delivery endpoint with it; the service watches the broker and pushes each message to the right endpoint.

Registration would carry what the adapter needs, e.g.:

• Kimi → local REST base URL + session id (discovered from ~/.kimi-code/session_index.jsonl) + bearer
• Codex → app-server WebSocket + thread id
• agy → agentapi endpoint
• OpenCode → deliver-watch/sidecar target

Why this shape:

• We already have the pattern in-repo. The relay connector is a machine-wide supervised singleton with a machine lock and self-heal (B200/B235, c2c start relay-connect, ocaml/cli/c2c_relay_managed.ml). Reuse it rather than inventing new infra — one supervised thing to keep alive, restart, and diagnose.
• The broker is already the machine-wide rendezvous for sessions and mail; endpoint registration is a natural extension of registration we already do (alias, session_id, client_type, cwd, pid).
• Reaping comes for free. A service that knows every registered endpoint can drop watchers for sessions that are gone — which is exactly the inverse-leak fix, instead of a separate reaper.
• One doctor surface. Today delivery health is checked per-client (kimi DEAF, codex delivery mode, grok drift); a single service gives one honest "is this session reachable?" answer.
• Clients needing wrapping converge. Codex, Kimi and agy all need an external process to reach them; that's the same job three times.

Scope / open questions

• Lifecycle: who starts it — c2c install, first c2c start <client>, or on demand? How does it survive a reboot (systemd --user is a portability tradeoff; the relay-connect supervisor pattern is the in-repo alternative).
• Migration from the per-alias C2c_kimi_notifier pidfile/sidfile lifecycle (see kimi notifier strands pre-startup inbox backlog (silent for senders) #9 — the alias-keyed state vs session-id-keyed inbox asymmetry is load-bearing there).
• Registration transport: extend broker registration, or a separate endpoints table?
• What "guaranteed" means operationally, and the failure modes that remain (host off, client's own server down, session id unresolvable, agent mid-turn).
• Must preserve B098 "bus, never RPC": delivery stays DATA-only; a message may never authorize an action or resolve an approval.

Related

#9 (Kimi notifier arming — the immediate reliability patch), #27 (managed Codex silently deaf), #31 (degraded signal overwritten), #12 (pre-startup backlog visibility). A design pass covering per-client wake-path inventory + this singleton is in progress.

[XertroV]

Design pass complete — with two corrections to the framing above

Correction 1 (safety-critical): Codex must NOT move into the singleton

Codex's gated auto-turn is the one narrowly sanctioned B098 scheduling effect, and it is correctly lifecycle-bound to its session (c2c_codex_deliver_loop.mli:15-25, gate in c2c_codex_autoturn.mli:41-46: local-broker mail only, thread idle, DND off; remote / @host / # senders fail closed). A machine-wide process able to start Codex turns would widen that gate to remote mail and delete the invariant. Kimi and agy belong in the singleton — both are plain external POSTs with no turn-scheduling authority. Codex's wake stays in-session.

Correction 2 (honest limit): Claude Code and Grok cannot be guaranteed from inside c2c

Neither exposes a local control surface that accepts a synthetic user turn. The three available paths are all disqualified: c2c monitor requires the agent to arm it (a model decision, so never a guarantee); the MCP dev channel sits behind an approval-gated experimental flag; PTY/tmux keystroke injection is deprecated and too fragile to found a guarantee on. Upstream ask (worth filing against both): a local authenticated endpoint that accepts an injected user turn on a running session — the shape Kimi/Codex/OpenCode already have — or a hook event that fires on external file change / inbound notification and can resume an idle session.

Per-client wake inventory (a "wake" = external push, no model decision)

Client	Mechanism	Wakes an IDLE agent?	Verdict
OpenCode	in-process plugin promptAsync, driven by session.idle + a background interval	Yes	GUARANTEED (while client runs)
Codex (managed/app-server)	thread/inject_items + gated auto-turn	Yes, local mail (DND off, thread idle)	GUARANTEED (local) / NONE (remote)
Codex (vanilla hooks)	activity-triggered hooks only	No	NONE
Kimi	external REST POST /api/v1/sessions/{id}/prompts	Yes	CONDITIONAL (daemon alive ∧ kimi server up ∧ session_index resolvable)
agy	agy agentapi send-message from the deliver loop	Yes	CONDITIONAL (loop alive ∧ agy-env resolvable)
Claude Code	PostToolUse / Stop / SessionStart hooks; gated MCP channel	No — all activity-triggered; mail arriving after Stop never fires it	NONE (CONDITIONAL only if the agent armed c2c monitor)
Grok	skill + SessionStart/SessionEnd hooks	No — the skill instructs the agent to arm a monitor; that's a model decision	NONE (CONDITIONAL if armed)

The structural insight

• Class A — in-process wake (OpenCode plugin, Codex deliver loop bound to session lifetime): cannot orphan, cannot outlive its session, needs no supervision. If it's dead, the client is dead and there's nothing to wake. No gap.
• Class B — out-of-process wake (Kimi notifier, agy loop, c2c monitor, deliver-inbox): lifetime decoupled from the session, so it fails in both directions — it dies (missed wake, the kimi notifier strands pre-startup inbox backlog (silent for senders) #9 DEAF case) and it outlives (the inverse leak seen since Jul 18).

So the move is not "supervise the per-alias daemons better" — it's collapse all of Class B into one supervised machine-wide process, exactly the relay-connect pattern (B200/B235).

And deliver-watch is not a wake mechanism — poll_once_kimi (c2c_deliver_inbox.ml:352-370) literally calls C2c_kimi_notifier.run_once, i.e. the same REST POST. inotify only changes how the external process learns mail arrived; it doesn't remove the external process. It must not be presented as a delivery guarantee anywhere in the docs.

Recommendation

Primary: a machine-wide supervised wake singleton modeled 1:1 on C2c_relay_managed — machine lock at ~/.local/share/c2c/wake, supervisor with restart-on-nonzero-exit and binary-stamp restart (so just install-all takes effect — this subsumes the per-alias B145 hand-rolling), daemonize + ready-pipe handshake, child enumerating all brokers (C2c_repo_fp.scan_repos_dir) + the sessions broker. Per-client adapters (kimi → C2c_kimi_deliver.deliver_message; agy → agentapi; everything else → an explicit No_wake_surface the doctor renders).

Backstop A (optional, never primary): systemd --user unit behind c2c install self --systemd, gated on systemctl --user, silent fallback elsewhere — it's the only thing covering reboot/logout, but mandating it breaks non-systemd hosts and containers (which our own docker-tests exercise).
Backstop B: opportunistic healing — any broker-touching c2c invocation does one cheap try_acquire; Already_running is a single syscall meaning "healthy, do nothing." Much cheaper than today's doctor --rearm (which scans registrations and forks per alias).

Reaping comes free: there is no per-alias process left to leak, plus a one-shot stop_all_daemons () at singleton startup reaps legacy survivors.

Load-bearing refactor: deliver_via_rest / run_once resolve the Kimi session from Sys.getcwd (). A machine-wide process has one cwd, so workdir must be threaded explicitly from each registration's cwd field (already captured for the worktree-mismatch guard). Keep current signatures as wrappers defaulting to getcwd so existing callers compile during migration.

Migration (no flag day): don't delete ensure_daemon — add an early return when try_acquire on the wake lock reports Already_running, so the singleton owns delivery and the alias-keyed pidfile/sidfile machinery from #9 becomes dead-but-harmless during the overlap.

What "guaranteed" actually buys

Covered: daemon crash/OOM/SIGKILL (supervisor restart ~1s); stale binary after install; never-armed — the #9 bug class is deleted, not mitigated (nothing is per-session, so nothing can fail to arm); sessions appearing mid-flight; the inverse leak.
Only with the systemd backstop: host reboot, user logout.
Never covered (must be documented): host powered off; client-local server down (best-effort restart only); session id not yet written to session_index.jsonl — a delay, not a loss, since undelivered mail is written back and retried; agent mid-turn — the POST queues as the next user turn, which is correct.

Suggested split (this issue becomes the P0)

1. P0 — the wake singleton (this issue): c2c_wake_managed.ml + adapter seam + kimi adapter + workdir refactor + ensure_daemon deferral.
2. P1 — legacy daemon reaping / inverse leak: small, independently landable, could ship first as a c2c doctor reaper to stop the bleeding.
3. P1 — wake-capability matrix in c2c doctor: render the table above per registered session (GUARANTEED / CONDITIONAL-with-condition / NONE). This is the product-visible artifact that stops the docs over-claiming for Claude and Grok.
4. P2 — upstream tracking: "Claude Code and Grok have no external idle-wake surface", plus correcting any website/runbook copy implying deliver-watch or PostToolUse constitutes a wake.

Invariants for whoever implements: DATA-only (no verdict path in the singleton, ever); never sweep, mutate registrations, or signal client processes — read inbox + write back undelivered only; respect metadata_opt_out and never log message bodies; per-adapter exception isolation so one bad session can't kill the loop.

[XertroV]

Implementation refinement — one hard blocker, and the teardown rules

BLOCKER: Sys.getcwd () must be removed from the delivery path first → filed as #36

Kimi session resolution is cwd-derived (c2c_kimi_notifier.ml:294 in deliver_via_rest, :471 in run_once). That works today only by accident: the per-alias notifier is forked from the session's own process and inherits its cwd. A machine-wide singleton has exactly one cwd — and the relay-connect pattern it copies does chdir "/" (c2c_relay_managed.ml:399) — so every watched session would resolve to the same wrong session id. #36 must land before this issue.

Teardown rules (these are the hard part — get them wrong and we recreate #9)

The inverse leak exists because a watcher is a process that must be killed by a lifecycle event that may never fire. Under the singleton a watcher is just an entry in a watch set rebuilt every tick from live registrations — a vanished session simply stops being in the set. There is no process to orphan; the Jul-18 daemon becomes structurally impossible. But:

1. Stop-watching is LOCAL and NON-DESTRUCTIVE. Dropping a session from the watch set touches the supervisor's own memory only. It must never sweep, drop a registration, or drain/delete an inbox. Mail for a vanished session stays queued and is delivered if it returns. (This is exactly the sweep-drops-managed-sessions mistake — the hard line.)
2. Liveness must fail OPEN, not closed. Drop a session only on confirmed death (registration gone / marked dead). Unknown, unreadable, or transiently-missing → keep watching. Wrongly dropping a live session recreates the DEAF bug kimi notifier strands pre-startup inbox backlog (silent for senders) #9 just fixed; wrongly keeping a dead one costs one cheap read_inbox per tick. The asymmetry is deliberate.
3. Supervisor shutdown reuses stop_supervisor (c2c_relay_managed.ml:365) — SIGTERM then SIGKILL after grace, with Fun.protect releasing the lock and pidfile.

On the inverse leak: do NOT file or fix it separately

It closes here, by construction, plus a one-shot stop_all_daemons () (c2c_kimi_notifier.ml:931, already identity-gated via pid_is_our_notifier so a reused PID is never signalled) at supervisor startup to retire legacy per-alias daemons. A standalone reaper would be throwaway work this issue deletes.

Other implementation notes

• Child tick (~2s): enumerate brokers (C2c_repo_fp.scan_repos_dir) → registrations → filter to externally-woken clients (is_kimi_registration, c2c_doctor_hooks.ml:762; add an agy predicate) → explicitly exclude claude/codex/opencode (their delivery is client-owned; double-delivery would be a bug) → reconcile watch set → run_once ~workdir.
• Keep it single-threaded and sequential at first, but add a hard per-session timeout and per-session error backoff: run_once does a blocking Lwt_main.run HTTP call, so a hung Kimi server would head-of-line-block every other session. Don't reach for threads until measurement demands it.
• Opportunistic arming (option 3) is the bootstrap, not the guarantee. The singleton lock makes ensure_supervisor () free to attempt anywhere (second attempt → Already_running, exit 0). Call it at every site that calls ensure_daemon today (c2c_start.ml:5199, c2c_hook_cmd.ml:1467, c2c_doctor_hooks.ml:889, install). Net effect is a large deletion — the per-alias SHA/pidfile/re-key logic collapses into one idempotent call. Its unique value is covering the post-reboot cold start (the one thing the rejected systemd option would have bought).
• Migration, no flag day: keep ensure_daemon as fallback; add the early return when the wake lock reports Already_running; make the sidecar pid tracking (c2c_start.ml:5203, teardown :4602/:5891) no-ops that unlink stale files; delete start_daemon's loop only once live-proven (run_once survives as the per-tick unit).
• c2c doctor must gain a wake-supervisor check (mirror c2c_doctor_relay.ml): supervisor alive, watch-set size, per-session last-successful-delivery age. A silent supervisor is worse than today's status quo — this is now a single point of failure for all Kimi/agy wake on the host, and that trade is only acceptable if the failure is loud.

Acceptance test (the gate)

Hermetic covers most of it (pure decide_watch_set; supervisor lock/crash/binary-change restart modelled on test_c2c_relay_managed.ml; REST via C2C_KIMI_DELIVER_FIXTURE; the #36 cwd RED test; a B098 regression proving a message body containing allow writes no verdict file). But per "if it is not tested in the wild it is not done", the gate is: kill -9 the wake child, confirm the supervisor relaunches and a message sent during the gap is delivered. Nothing else proves the guarantee. Plus two live Kimi sessions in different repos to prove per-session workdir resolution.

Revised split

1. Thread explicit ~workdir through Kimi delivery (remove Sys.getcwd) — blocks the wake service #36 — workdir refactor. Blocks this. Land first, separately.
2. This issue — the wake singleton (+ opportunistic arming + doctor check + legacy stop_all_daemons sweep). The inverse leak closes here.
3. agy extension — same structural gap, but let the Kimi path prove the design first.

Also filed: #37 — Grok has no guaranteed wake at all (contingent on the agent arming a monitor); not fixable by this supervisor, needs an upstream surface.

[XertroV]

Status for auto-delivery goal (2026-07-20): remains OPEN as north-star design (#35 machine-wide delivery service). Not blocking per-client primary paths (plugin / app-server / REST / agentapi). Tracking separately from send-keys elimination.

[XertroV]

Phase 1 scaffold landed on local master (merge 15def348).

• c2c start|restart|stop deliver-service — machine-wide supervised singleton (lock + pidfile), idle loop only
• c2c doctor deliver-service [--json] — alive/dead stub
• Hermetic suite test_c2c_deliver_managed (lock exclusive, crash release, Already_running, clean stop)
• No adapters yet — per-client notifiers unchanged

Phase 2 plan: .collab/plans/2026-07-20-deliver-service-phase2-kimi.md (kimi adapter alongside notifier). Issue stays OPEN as the epic until adapters + live kill-9 acceptance.

[XertroV]

Phase 2 (kimi adapter) landed on local master.

Flag-gated dual-run with per-alias notifier (default off):

• C2C_DELIVER_SERVICE_KIMI=1 + C2C_DELIVER_SERVICE_KIMI_MODE=shadow|active|primary
• shadow (default): rebuild watch-set, log would-deliver, no POST
• active: POST via run_once only when already_running alias is false (DEAF fallback / single-writer)
• primary: service always POSTs for watched rows
• Fail-open scan merge + delayed drop; workdir required (Thread explicit ~workdir through Kimi delivery (remove Sys.getcwd) — blocks the wake service #36)
• Hermetic: test_c2c_deliver_managed 16/16

Still OPEN until live acceptance (plan L1–L6, especially kill-9 gap tests) and multi-day dogfood. Phase 3 = migrate off per-alias daemon defaults.
Plan: .collab/plans/2026-07-20-deliver-service-phase2-kimi.md

[XertroV]

P3 architecture deepen (C2→C3→C1) landed on local master (I014 / backlog P3).

Slice	Module	Status
C2	C2c_kimi_delivery_claim	claim-before-POST, both loops, hermetic 6/6
C3	C2c_liveness	purpose-tagged facade, doctor pid_alive shared
C1	C2c_delivery_endpoint	hybrid registry + kimi adapter; Mode_* still temporary

#35 remains OPEN for live kill-9 dogfood + multi-client adapters + Mode_* retirement (≥2 adapters rule).

[XertroV]

P4 residual slice landed (local master):

• 2nd adapter: agy via C2c_agy_agentapi + C2c_delivery_endpoint.Agy (probe agy-env, agentapi deliver). Registry now kimi+agy (≥2 → Mode_* retirement eligible, not done).
• Multi-client watch: C2C_DELIVER_SERVICE_AGY=1 alongside kimi flags; Active agy uses adapter deliver+drain.
• Live smoke: deliver-service start (kimi shadow) / doctor JSON / stop — finding 2026-07-20T13-15-00Z-deliver-service-p4-smoke.md.

Still OPEN for #35 close: multi-day Active dual-run + kill-9 gap (L4/L5) with live managed sessions; Mode_* collapse; full agy wake e2e.