fix(cli): shut down in order on ctrl-c, keeping the bus alive throughout - #280
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #280 +/- ##
==========================================
- Coverage 52.03% 51.97% -0.07%
==========================================
Files 102 102
Lines 8614 8626 +12
Branches 997 999 +2
==========================================
+ Hits 4482 4483 +1
- Misses 3924 3934 +10
- Partials 208 209 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
wschoenell
approved these changes
Jul 28, 2026
A ctrl-c used to raise KeyboardInterrupt inside the selector loop on the main thread, tearing the bus down mid-unwind before any orderly shutdown could run: in-flight RPCs died with BusDeadException, instrument control loops crashed resolving proxies, and manager.shutdown() ran against a dead bus. Install a SIGINT handler that never raises: it spawns a daemon shutdown thread and select() resumes (PEP 475), so the bus keeps routing local and remote traffic while the manager stops objects; the bus goes down last through its normal tested path. A second ctrl-c force-exits with status 130.
phsilva
force-pushed
the
graceful-shutdown-cli
branch
from
July 29, 2026 20:51
90319ec to
12cd5c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of two stacked PRs fixing the Ctrl+C shutdown cascade. This one fixes the
root cause in the CLI; the scheduler-side fixes that make the in-flight program
abort cleanly are stacked on top in #279.
Problem
Ctrl+C on a running
chimeraproduced a cascade of tracebacks: in-flight RPCsdied with
BusDeadException, instrument control loops crashed resolvingproxies, and
manager.shutdown()only ran after everything had already burneddown.
Root cause: the bus selector loop runs on the main thread, so SIGINT became a
KeyboardInterruptinsideselector.select()that unwound through_run'sfinallyand ran_teardown()mid-unwind — the bus was dead and everymailbox closed before any orderly shutdown could begin. On top of that,
ChimeraCLI.shutdown()stopped the bus before the manager, so even theorderly path ran object teardown against a dead bus.
Design
Never let SIGINT break the selector loop. The CLI installs a handler that never
raises: it spawns a daemon shutdown thread and
select()simply resumes(PEP 475). The bus keeps routing — local and remote (remote responses arrive
on the socket only the selector loop reads, and proxies default to no timeout,
so shutdown RPCs need it alive) — while the manager stops objects newest-first.
Only then does
bus.shutdown()run, through its normal tested path: the wakerfd wakes the selector, the loop thread runs
_teardown(), andrun_forever()returns. A second Ctrl+C force-exits with status 130.
Zero changes to bus.py; both existing bus-shutdown tests are untouched
(
test_bus_graceful_shutdownalready covers exactly the path Ctrl+C nowtakes).
Verification
instruments stopped in order, bus last, exit 0 with zero tracebacks; SIGINT
while idle → exit 0; double SIGINT → forced exit 130