fix(manta-cli): MAN-85 — drain in-flight spots on SIGTERM (and SIGHUP) the same way SIGINT already does - #127
Open
catalyst-cloud-connector[bot] wants to merge 1 commit into
Open
fix(manta-cli): MAN-85 — drain in-flight spots on SIGTERM (and SIGHUP) the same way SIGINT already does#127catalyst-cloud-connector[bot] wants to merge 1 commit into
catalyst-cloud-connector[bot] wants to merge 1 commit into
Conversation
… same way it already does on SIGINT
catalyst-cloud-connector
Bot
force-pushed
the
MAN-85
branch
from
September 7, 2026 05:10
02a8f12 to
bb00b58
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
manta listenonly registered a handler for SIGINT (ctrlc::set_handler,crates/manta-cli/src/main.rs:1170, gated by thectrlccrate's default feature set). Every real service manager — systemd, Docker, Kubernetes — sends SIGTERM by default on stop, not SIGINT. With no handler registered, SIGTERM kept the OS default disposition and the kernel killed the process directly: measured on this branch's base commit, plainlistenexited in 11 ms with exit code 143 (raw signal), and a server-mode run with a connected, logged-in telnet client and an unread backlog behaved identically — zero drain-related log output, process gone. Only Docker'sSTOPSIGNAL SIGINTworkaround masked this in the one deployment path that had it.Fix
Enable
ctrlc'sterminationfeature, which registers SIGINT, SIGTERM and SIGHUP behind the same handler closure. No change tomain.rs's shutdown logic was needed: the existing handler already treats "the registered signal fired" as one undifferentiated event (it just flips anAtomicBool), so oncectrlcforwards SIGTERM to that closure it flows into the exact samestop-flag →shutdown_tx.send(true)→ per-client drain →tasks::await_all(SHUTDOWN_DRAIN_DEADLINE)→Ok(())/exit 0 sequence SIGINT already used.Cargo.lockis unaffected —termination = []has no dependencies of its own.A
tokio::signal::unix::signal-based alternative (also suggested by the ticket) was considered and rejected: this codebase constructs atokio::runtime::Runtimein exactly one place (start_spot_server, reached only when--server-configis given), so a plainmanta listen --source foo.wavhas no reactor to poll atokio::signalfuture. Adopting it would have meant either standing up a runtime solely to host a signal task, or splitting signal handling into two divergent paths for server vs. non-server mode — the opposite of the ticket's goal that both signals share one code path. Thetermination-feature fix has no such asymmetry.Accepted consequence: SIGHUP now means "shut down"
ctrlc'sterminationfeature is all-or-nothing — its public API has no way to select a subset of {SIGINT, SIGTERM, SIGHUP}. SIGHUP previously shared SIGTERM's bug (OS-default kill, no drain); it's now a strict improvement, and is pinned by a dedicated test rather than left as an accidental side effect. This does foreclose one specific future design: the 2026-09-05 broad review's R-08 ("live config reload via SIGHUP", cross-referenced to MAN-30) can no longer claim SIGHUP throughctrlc's simple API — it would need its own signal-distinguishing mechanism, or themanta reloadsubcommand form instead. Recorded in the decision doc below so this isn't rediscovered later.What changed
Cargo.toml—ctrlc = "3"→ctrlc = { version = "3", features = ["termination"] }(the actual fix, one line).crates/manta-cli/src/main.rs— oneeprintln!immediately afterctrlc::set_handler(...)?, printing a stderr-only readiness marker ("manta: listening; send SIGINT or SIGTERM to stop").listenpreviously printed nothing at startup, so there was no way for a test (or an operator) to know the signal handler was actually installed before sending a signal; this closes that gap and doubles as the new test's readiness handshake. Confirmed stdout stays pure JSON under--json(MAN-59's stdout-purity contract) — the marker is stderr-only and the byte-for-byte stdout output is unchanged versus the pre-change binary.crates/manta-cli/tests/signal_shutdown.rs(new) — four subprocess-spawning integration tests that send real OS signals to a realmanta listenchild and assert on its actual exit status, since nothing in this repo previously exercised real signal delivery and no#[cfg]in this crate can observe a dependency's feature flag:sigterm_exits_zero_through_the_drain_path/sigint_exits_zero_through_the_drain_path/sighup_exits_zero_through_the_drain_path— spawnmanta listenagainst a 300 s 48 kHz fixture (long enough that natural EOF can't be confused with the signal exiting the process — a concurrently-run, unsignalled "control" child guards this), wait for the readiness marker, signal, and assertstatus.code() == Some(0)within a bounded window. A signal-killed process can never satisfy that assertion (code()isNonewhen killed), so the lock can't pass by accident.dockerfile_does_not_retarget_the_container_stop_signal— asserts the Dockerfile carries noSTOPSIGNALdirective, so the workaround this PR removes can't silently come back.crates/manta-cli/Cargo.tomlgainslibcas a dev-only dependency (already a workspace dependency) to calllibc::killdirectly, rather than shelling out to/bin/kill.Dockerfile— removedSTOPSIGNAL SIGINTand its now-inaccurate justification comment; replaced with a comment explaining why Docker's default stop signal is correct again post-fix. The separate grace-period comment (aboutdocker stop -t 30— Docker's 10 s default grace period being shorter thanSHUTDOWN_DRAIN_DEADLINE) is retained with one sentence reworded, since that guidance is about drain duration, not which signal triggers it, and is unaffected by this change.README.md's matchingdocker stop -t 30guidance was left untouched for the same reason.docs/DECISIONS/2026-09-07-man85-signal-handling.md(new) — records the root cause, the two options considered, the decision, and the SIGHUP consequence for R-08/MAN-30, per this repo'sdocs/DECISIONS/convention.Verification
Reproduced and fixed live in the implementing container (not narrated), and independently re-verified end-to-end during validation:
listen, SIGINTlisten, SIGTERMlisten, SIGHUPterminationfeature reverted, SIGTERMFull workspace gate, run against the completed change:
cargo fmt --all --checkclean;cargo clippy --workspace --all-targets --locked -- -D warningsclean, 0 warnings;cargo test --workspace --locked— 470 passed, 0 failed, 9 pre-existing ignored (hardware/soak gates, unrelated). The newsignal_shutdowntarget: 4/4 passing.A
/code-reviewpass at high effort over the full diff found zero correctness or security findings, and zero findings outside the new test file. Five test-robustness findings (a fixture cache that doesn't invalidate on parameter change, an exit-time budget looser than the fixture's measured natural EOF, no kill-on-drop for spawned children on panic paths,await_readyclosing stderr and discarding later diagnostics, a case-sensitiveSTOPSIGNALgrep guard) were identified incrates/manta-cli/tests/signal_shutdown.rs. None weakens the regression lock's actual guarantee — the primary assertion (status.code() == Some(0), unsatisfiable by a signal-killed process) catches the real regression deterministically regardless of timing, as proven by the revert-and-rerun above. Per this repo's round-2+ review-convergence policy, these are captured for a follow-up ticket rather than fixed inline here:Not in scope (deliberately)
SHUTDOWN_DRAIN_DEADLINE's shape (MAN-45) — a pre-existing property of the SIGINT drain path (one flat 25 s budget across all tracked tasks). This PR only makes SIGTERM reach that path; it doesn't change its behavior.Command::Soak— registers no signal handler and stops on its own duration watchdog; it's a bounded self-test, not a long-running daemon.packaging/(MAN-75, in flight, not yet merged as of this branch's base) — shipsmanta.servicewithKillSignal=SIGINT, a launchdkill -INTworkaround, and adocker-compose.ymlcomment referencing the now-removedSTOPSIGNAL SIGINT. All three stay correct (not wrong) after this PR regardless of merge order, since SIGINT still works identically — they just become redundant. Follow-up once MAN-75 lands: dropKillSignal=SIGINTand its matching test assertion from the systemd unit, drop the launchdkill -INTworkaround, and update the staledocker-compose.ymlcomment. This list is also captured in the committed decision doc so it isn't lost.dockernor a live systemd are available in the container this was built and validated in. The four newsignal_shutdowntests exercise the same underlying code path natively; a manualdocker build && docker stop -t 30 <container>run is still recommended before the next release tag. The code review did confirm the Dockerfile'sENTRYPOINTis exec-form (not shell-form), which is the property that makes removingSTOPSIGNALsafe —mantaruns as PID 1 and genuinely receives the signal Docker sends.References
MAN-85, found via the 2026-09-05 broad review (lens 1 #4, lens 2 #5). Related, not duplicated: MAN-45 (undersized per-client drain deadline within the existing SIGINT path), MAN-30/R-08 (future SIGHUP-triggered config reload, now constrained by this PR's SIGHUP consequence), MAN-75 (in-flight packaging ticket whose
KillSignal=SIGINT/kill -INTstopgaps this PR supersedes).