Skip to content

feature: verified Linux install and systemd service lifecycle #945

Closed

Description

@huang195

Feature Description

Same goal as the macOS install issue, on the systemd path, which shares none of launchd's code.

Proposed Solution

Additional Context

Part of the Cortex v0.9.0 release. Gates exit criterion 1 (install works cold). Automated by
the Linux release smoke test.

Activity

  1. added this to the Release V0.9.0 milestone on Sep 10, 2026
  2. Alan-Cha commented on Sep 23, 2026

    @Alan-Cha
    Member

    Status update from the #1076/#1079/#1080 (+follow-up #1107) review cycle, full analysis in authbridge/docs/superpowers/specs/2026-09-16-linux-systemd-lifecycle-design.md:

    Closed:

    Partial, not checking off yet:

    Untouched:

    • Fresh install via curl \| sh on a real Linux box, and upgrade-over-existing against a real running unit — both need actual CI infrastructure, likely belongs with test: release smoke test on Linux CI runners #957 (Linux release smoke test) rather than this issue.
    • No Linux equivalent of launchdUsable()'s clean preflight — explicitly left as an open decision in the design doc, not yet resolved either way.
  3. Alan-Cha commented on Sep 23, 2026

    @Alan-Cha
    Member

    #1108 addresses the last untouched bullet — "Works under user systemd, and states what happens where systemd is absent" — by adding systemdUsable() (the Linux equivalent of launchdUsable()) and wiring it into serviceInstall's existing preflight. Not checking the box until it merges.

  4. Alan-Cha commented on Sep 29, 2026

    @Alan-Cha
    Member

    #1108 merged — checking off "Works under user systemd, and states what happens where systemd is absent."

    Unrelated but worth flagging here: main has since undergone a large reorg (afb49e9f, "Flatten authbridge/ into the repo root") — authbridge/cmd/abctl/... is now just cmd/abctl/..., authbridge/authlib/ is now core/, etc. Our code landed before that and moved automatically with it, but the design doc's own path references (authbridge/cmd/abctl/cmd_service_platform.go etc.) still say authbridge/... and are now stale in prefix. Not fixing preemptively — flagging so whoever picks up the remaining bullets knows to use the new paths, and can fix the doc's prefixes as part of that work rather than as a separate no-op PR.

    Remaining, for reference:

    • Bullets 1-2 (fresh install + upgrade against real hardware) really belong to test: release smoke test on Linux CI runners #957 (Linux release smoke test CI), not this issue.
    • Bullet 3's remainder (full serviceUninstall end-to-end: unit file, stamp file, ~/.cortex actually removed) and bullet 6's remainder (a real-systemd integration test that drives loadService/controlService/supervisorRunning themselves, not just raw systemctl — the actual analog of TestWaitBootedOut_RealLaunchd) are the two pieces of actual #945-scoped work still open.
    • Bullet 5's reboot half is tracked in test: verify the installed Cortex service survives reboot on macOS and Linux #964, not here (CI runners don't reboot).
  5. Alan-Cha commented on Oct 7, 2026

    @Alan-Cha
    Member

    Checked off the upgrade box — now covered end-to-end by the Linux release smoke test (#957, landed in #1288): assert_running_binary_is_current + assert_running_version_is confirm the systemd unit is actually running the newly-installed binary post-upgrade (not a stale process, the exact #1203-class regression this is guarding against), and a config marker confirms the config survives untouched.

  6. Alan-Cha commented on Oct 8, 2026

    @Alan-Cha
    Member

    Split and corrected the uninstall bullet: the original wording bundled 'no unit file, no ~/.cortex, no modified agent settings' as one guarantee. In practice:

    • unit file removal + leaving ~/.cortex (config, CA) untouched is done and tested (checksum comparison in the Linux release smoke test) — checked that half.
    • reverting agent settings is NOT automatic on uninstall. serviceUninstall explicitly tells the user to separately run agentop configure claude-code disable / agentop configure opencode disable — left that half unchecked and split out as its own line, since it's a real gap (or a deliberate scoping decision someone should confirm), not a wording issue. Possibly belongs with feature: agent integration contract — reusable detection, settings patching, and revert #939's settings-revert work.
  7. Alan-Cha commented on Oct 15, 2026

    @Alan-Cha
    Member

    #1405 adds the arm64 half of "Fresh install via curl | sh, amd64 and arm64" — matrix-izes the Linux release smoke test (#957) to also run on GitHub's ubuntu-24.04-arm runner, exercising the exact same fresh-install/upgrade/uninstall path install.sh's own uname -m detection and release-binaries.yaml's linux/arm64 tarballs already support. Not checking the box until it merges.

  8. Alan-Cha commented on Oct 28, 2026

    @Alan-Cha
    Member

    Checked off three more boxes:

    Fresh install via curl | sh, amd64 and arm64. #1405 (merged 2026-10-15) runs the Linux release smoke test on both ubuntu-24.04 and ubuntu-24.04-arm. It installs the way curl | sh does: it downloads install.sh and feeds it to sh -s on stdin, so the script's own re-exec bootstrap behaves as it does for a real user. The latest run passed on both arches: https://github.com/rossoctl/cortex/actions/runs/40712931917

    Uninstall reverts agent settings automatically. When I split this line out on 2026-10-08, uninstall did not revert anything. That has since changed: agentop uninstall (#1252, merged 2026-10-03) unroutes Claude Code, OpenCode, IBM Bob and the bob shell function before it stops the service, and leaves a key alone if the user has edited it since enable. Its tests cover each agent (TestUninstallUnroutesOpenCode, TestUninstallUnroutesBobAndTheBobShell, TestUninstallLeavesAClaudeCodeKeyChangedSinceEnable, …). I reworded the line to say which command it means, because there are two:

    • agentop uninstall removes Cortex: agents unrouted, service removed, binaries and PATH lines removed, ~/.cortex kept unless --purge.
    • agentop service uninstall removes only the service. It warns that Claude Code and OpenCode fail while the proxy is down and names the configure … disable commands, but changes no agent settings. That is deliberate, since it is the command for managing the service, not for removing Cortex.

    Service survives reboot and a crash.

    Still open: agentop service status | start | stop | restart accurate in every state. Checking it on a real host showed it is not:

    State systemd agentop service status exit
    running active Cortex is healthy 0
    stopped with service stop inactive Cortex is NOT answering … Claude Code and OpenCode will fail 1
    crash-looping activating Cortex is NOT answering … Claude Code and OpenCode will fail 1

    A deliberate stop reads as a failure. I'm opening a PR that makes status ask systemd for the unit's state, and adds the real-systemd test this item has been missing: one that drives agentop service itself through install, status, stop, start, restart, a crash and uninstall, in CI's real-systemd job.

  9. Alan-Cha commented on Oct 28, 2026

    @Alan-Cha
    Member

    #1529 is the PR for the last unchecked item, "agentop service status | start | stop | restart accurate in every state". It makes status ask systemd why Cortex is silent, so a service stop reports as stopped (exit 0) rather than as a failure. It makes start/restart/install clear systemd's start limit, so they can recover a unit that crash-looped. And it adds TestServiceLifecycle_RealSystemd, which drives agentop service itself through every state against real systemd in CI. It passes on systemd 257 (Debian 13) and 261 (Ubuntu 26.10). I'll check the box once it merges.

  10. Alan-Cha commented on Nov 4, 2026

    @Alan-Cha
    Member

    The last box is done: #1529 (merged) makes agentop service status report each systemd state accurately: running, stopped by service stop, stopped some other way, restarting after a crash, and given up after a crash loop. start and restart now recover a unit that hit systemd's start limit, and TestServiceLifecycle_RealSystemd drives the whole lifecycle against a real systemd. I verified it on Debian 13 (systemd 257) and Ubuntu (261). One edge case from its review, stopping a unit after systemd gave up on it, is fixed in #1660. The reboot-survival work continues in #964. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions