Skip to content

fix(cli): shut down in order on ctrl-c, keeping the bus alive throughout - #280

Merged
phsilva merged 1 commit into
astroufsc:masterfrom
phsilva:graceful-shutdown-cli
Jul 29, 2026
Merged

fix(cli): shut down in order on ctrl-c, keeping the bus alive throughout#280
phsilva merged 1 commit into
astroufsc:masterfrom
phsilva:graceful-shutdown-cli

Conversation

@phsilva

@phsilva phsilva commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

First of two stacked PRs fixing the Ctrl+C shutdown cascade. This one fixes the
root cause in the CLI; the scheduler-side fixes that make the in-flight program
abort cleanly are stacked on top in #279.

Problem

Ctrl+C on a running chimera produced a cascade of tracebacks: in-flight RPCs
died with BusDeadException, instrument control loops crashed resolving
proxies, and manager.shutdown() only ran after everything had already burned
down.

Root cause: the bus selector loop runs on the main thread, so SIGINT became a
KeyboardInterrupt inside selector.select() that unwound through _run's
finally and ran _teardown() mid-unwind — the bus was dead and every
mailbox closed before any orderly shutdown could begin. On top of that,
ChimeraCLI.shutdown() stopped the bus before the manager, so even the
orderly path ran object teardown against a dead bus.

Design

Never let SIGINT break the selector loop. The CLI installs a handler that never
raises: it spawns a daemon shutdown thread and select() simply resumes
(PEP 475). The bus keeps routing — local and remote (remote responses arrive
on the socket only the selector loop reads, and proxies default to no timeout,
so shutdown RPCs need it alive) — while the manager stops objects newest-first.
Only then does bus.shutdown() run, through its normal tested path: the waker
fd wakes the selector, the loop thread runs _teardown(), and run_forever()
returns. A second Ctrl+C force-exits with status 130.

Zero changes to bus.py; both existing bus-shutdown tests are untouched
(test_bus_graceful_shutdown already covers exactly the path Ctrl+C now
takes).

Verification

@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.97%. Comparing base (d2f918a) to head (12cd5c5).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
src/chimera/cli/chimera.py 0.00% 15 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #280      +/-   ##
==========================================
- Coverage   52.03%   51.97%   -0.07%     
==========================================
  Files         102      102              
  Lines        8614     8626      +12     
  Branches      997      999       +2     
==========================================
+ Hits         4482     4483       +1     
- Misses       3924     3934      +10     
- Partials      208      209       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A ctrl-c used to raise KeyboardInterrupt inside the selector loop on the
main thread, tearing the bus down mid-unwind before any orderly shutdown
could run: in-flight RPCs died with BusDeadException, instrument control
loops crashed resolving proxies, and manager.shutdown() ran against a
dead bus. Install a SIGINT handler that never raises: it spawns a daemon
shutdown thread and select() resumes (PEP 475), so the bus keeps routing
local and remote traffic while the manager stops objects; the bus goes
down last through its normal tested path. A second ctrl-c force-exits
with status 130.
@phsilva
phsilva force-pushed the graceful-shutdown-cli branch from 90319ec to 12cd5c5 Compare July 29, 2026 20:51
@phsilva
phsilva merged commit 5599e42 into astroufsc:master Jul 29, 2026
1 check failed
@phsilva
phsilva deleted the graceful-shutdown-cli branch July 29, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants