Repository navigation
SIGTERM during -reindex / -reindex-chainstate is ignored or kills mid-batch: no handler until startup finishes #907
Description
Found by Start9's intake review of the StartOS package (Start9-Community/satd-startos#1, "Blocked on", item 2):
SIGTERM is ignored while connecting blocks.
docker stopwith a 600 s grace period ran the full 600 s and killed it. StartOS sends SIGTERM and kills after the package'ssigtermTimeout, so any stop, reboot or settings change during the reindex produces #1. Seen on the StartOS box too, on an ordinary sync.
Their "#1" is #906: an interrupted full -reindex starts over.
During a replay
satd has no SIGTERM handler while -reindex, -reindex-chainstate or -upgradechainstate runs:
- Where the replay runs. It runs synchronously during startup:
reindex_from_flat_filesatsatd/src/main.rs:1532,reindex_chainstateat:1563. - Where the handler is registered. The only SIGTERM handler on that path is registered after both, at
satd/src/main.rs:4415. The other one, infast_start.rs:130, is local to the snapshot download.
So SIGTERM gets its default disposition for the whole replay, which can mean either of two things:
- satd is PID 1 of its PID namespace, with no init in front of it. The kernel drops default-action signals sent to a namespace's init, so SIGTERM is ignored until the supervisor's SIGKILL. That matches the 600 s
docker stop. The published image putstiniin front of satd (Dockerfile:240), so this depends on how the container was started; a package that execssatdas its entrypoint gets this. - Anywhere else. SIGTERM kills the process at once, mid-batch, the same as SIGKILL.
Neither is a graceful stop. Either way the replay ends unclean and #906 then throws it away.
During an ordinary sync
The handler exists by then, and Start9 still saw the stop run out its timeout on a StartOS box. Not yet explained. It needs a measurement of how long shutdown takes after SIGTERM during IBD, and what it waits on: the connect thread finishing its batch, the final flush of a large dbcache, or something else.
Expected
- During a replay. SIGTERM (and SIGINT) stops satd at the next block boundary: it finishes the block it is connecting, flushes, leaves the rebuild marker in place, and exits 0. The time this takes must fit a supervisor's grace period. StartOS's
sigtermTimeoutand Docker's default 10 s are the reference points. - This holds whether or not satd is PID 1, so a package that runs satd without an init behaves the same as the published image.
- During IBD. A stop after SIGTERM completes within a stated bound, and the log shows what shutdown is waiting on while it waits.
Acceptance
- An integration test that sends SIGTERM partway through the connect phase of a
-reindexand of a-reindex-chainstate. satd exits 0 within the bound, the marker is still there, and the next start continues the rebuild (with An interrupted full -reindex starts over instead of resuming from the block files #906). - The same with satd as PID 1 of a new PID namespace (for example
unshare --pid --fork, or a container without--init). - SIGTERM during IBD of a regtest chain with large blocks: satd exits within the bound.
Activity
Stop a startup block replay cleanly on SIGTERM, also as PID 1 (#907)
Abandon the block in hand when a replay is asked to stop (#907)
Continue an interrupted -reindex from the block files on any start (#906
Abandon the block in hand when a replay is asked to stop (#907)
Continue an interrupted -reindex from the block files on any start (#906
Stop a startup block replay cleanly on SIGTERM, also as PID 1 (#907)
Abandon the block in hand when a replay is asked to stop (#907)
Stop a startup block replay cleanly on SIGTERM, also as PID 1 (#907) (#…
Continue an interrupted -reindex from the block files on any start (#906
Continue an interrupted -reindex from the block files on any start (#906
Stop a startup block replay cleanly on SIGTERM, also as PID 1 (#907) (#…
Continue an interrupted -reindex from the block files on any start (#906