test: verify the installed Cortex service survives reboot on macOS and Linux #964

Activity

1. huang195

   added this to the Release V0.9.0 milestone on Sep 11, 2026
2. huang195

   assigned

   Alan-Cha

   on Sep 11, 2026
3. huang195

   added

   high priorityMust be address ASAPMust be address ASAP

   on Sep 11, 2026
4. huang195

   added a parent issue on Sep 11, 2026

   • epic: Cortex v0.9.0 — stable local tool for coding agents #962
5. github-project-automation

   added this to Rossoctl Issue Prioritizationon Sep 11, 2026
6. github-project-automation

   moved this to New/ToDo in Rossoctl Issue Prioritizationon Sep 11, 2026
7. Alan-Cha commented on Oct 15, 2026

   

   Alan-Cha

   on Oct 15, 2026

   Last edited by Alan-Cha

   Member

   More actions

   Ran the Linux half of this manually against a real VM (alan@192.168.64.3, Ubuntu, main-latest), using a new script: scripts/dev/verify-reboot-lingering.sh, submitted in #1409.

   Why a script instead of just doing it by hand: this needs a real reboot with nobody logged in, which neither ci.yaml's agentop leg nor release-smoke-linux.yaml can exercise — both enable systemd lingering themselves before install.sh runs, so Cortex's own "turn lingering on and remember that I did" branch never fires there, and GitHub-hosted runners can't be rebooted anyway.

   What it checked, and the results:

   1. Fresh install claims lingering. Starting from Linger=no, installing Cortex turned lingering on and wrote the linger-enabled-by-agentop marker recording that it did so. ✅

   2. Survives a real reboot, timed. 5 real reboot cycles, each timed via monotonic clocks on both ends (ActiveEnterTimestampMonotonic for the user manager + cortex.service, /proc/uptime at first successful healthz) — no wall-clock skew to account for. Boot-to-healthy:

      cycle 1: 4.51s
      cycle 2: 4.62s
      cycle 3: 4.28s
      cycle 4: 4.53s
      cycle 5: 4.40s
      min=4.28s median=4.51s max=4.62s (n=5)
      

   3. Lingering-ownership edge cases (uninstall shouldn't clobber a preference it didn't set):

      • Uninstall turned lingering back off, since Cortex itself had enabled it. ✅
      • Installing onto a host that already had lingering on (simulating an unrelated pre-existing preference) did not write the ownership marker. ✅
      • Uninstalling from that state left the pre-existing lingering preference untouched. ✅

   Host was restored to its original Linger=no baseline afterward.

   This covers the Linux reboot-survival half of this issue. Not covered here: the macOS equivalent, unclean shutdown/power-cycle (vs. a clean reboot), and session/metrics persistence across the restart — happy to pick those up separately if useful.

   ---

   Updated 2026-10-21: the first version of this comment claimed more than its evidence showed. Its timings were taken after the script's own SSH login, which starts the user manager, and Cortex with it, whether lingering works or not (huang195's review on #1409). The script now checks the host's own records instead. The figures below come from that version, run on a different VM.

   Ran the Linux half of this manually against a real VM (Ubuntu 26.10, aarch64, systemd 261, main-latest), using scripts/dev/verify-reboot-lingering.sh from #1409.

   Why a script instead of doing it by hand: this needs a real reboot with nobody logged in. Neither ci.yaml's agentop leg nor release-smoke-linux.yaml can do that. Both enable systemd lingering themselves before install.sh runs, so Cortex's own "turn lingering on and remember that I did" branch never fires there, and GitHub-hosted runners can't be rebooted anyway.

   What it checked, and the results:

   1. Fresh install claims lingering. Starting from Linger=no, installing Cortex turned lingering on and wrote the linger-enabled-by-agentop marker recording that it did. ✅

   2. Survives a real reboot, through lingering. 5 real reboots. For each one, the script compared when user@1000.service and cortex.service became active with when the boot's first login session opened, both on the since-boot clock as the host recorded them. It stayed off the host for 30 s after each reboot so the two times are far apart:

      cycle 1: manager 3.74s, cortex 3.74s, probe 30.49s | first login 30.32s -> 26.58s margin
      cycle 2: manager 3.71s, cortex 3.71s, probe 27.51s | first login 27.35s -> 23.64s margin
      cycle 3: manager 3.70s, cortex 3.70s, probe 32.73s | first login 32.57s -> 28.86s margin
      cycle 4: manager 3.76s, cortex 3.76s, probe 32.81s | first login 32.67s -> 28.92s margin
      cycle 5: manager 3.77s, cortex 3.77s, probe 31.69s | first login 31.51s -> 27.74s margin
      
      boot -> cortex.service active        min=3.70s median=3.74s max=3.77s (n=5)
      boot -> probe answered (upper bound) min=27.51s median=31.69s max=32.81s (n=5)
      

      On every boot, Cortex was up 23.6–28.9 s before anyone logged in, so lingering started it. ✅ The first summary line is the figure to quote. The second is when the script's health probe first got an answer, which can't happen before the 30 s wait ends.

   3. Lingering-ownership edge cases (uninstall shouldn't undo a preference it didn't set):

      • Uninstall turned lingering back off, since Cortex itself had enabled it. ✅
      • Installing onto a host that already had lingering on did not write the ownership marker. ✅
      • Uninstalling from that state left the pre-existing lingering preference untouched. ✅

   The host was restored to its original Linger=no baseline afterwards.

   This covers the Linux reboot-survival half of this issue. Not covered here: the macOS equivalent, unclean shutdown or power loss (as opposed to a clean reboot), and session/metrics persistence across the restart. Happy to pick those up separately if useful.
8. Alan-Cha

   added a commit that references this issue on Oct 21, 2026

   Test: Prove the reboot came up through lingering, not a login (#964)

   ...

   19851c0
9. evaline-ju

   added a commit that references this issue on Oct 28, 2026

   Test: Add a real-reboot verification script for Linux lingering (ross…

   ...

   2d28851
10. mrsabath

    removed a parent issue on Nov 2, 2026

    • epic: Cortex v0.9.0 — stable local tool for coding agents #962
11. mrsabath

    added a parent issue on Nov 2, 2026

    • epic: Remote Cortex — running Cortex in remote locations rossoctl#2651