Skip to content

SIGTERM during -reindex / -reindex-chainstate is ignored or kills mid-batch: no handler until startup finishes #907

Closed

Description

@bkeroack

Found by Start9's intake review of the StartOS package (Start9-Community/satd-startos#1, "Blocked on", item 2):

SIGTERM is ignored while connecting blocks. docker stop with a 600 s grace period ran the full 600 s and killed it. StartOS sends SIGTERM and kills after the package's sigtermTimeout, so any stop, reboot or settings change during the reindex produces #1. Seen on the StartOS box too, on an ordinary sync.

Their "#1" is #906: an interrupted full -reindex starts over.

During a replay

satd has no SIGTERM handler while -reindex, -reindex-chainstate or -upgradechainstate runs:

  • Where the replay runs. It runs synchronously during startup: reindex_from_flat_files at satd/src/main.rs:1532, reindex_chainstate at :1563.
  • Where the handler is registered. The only SIGTERM handler on that path is registered after both, at satd/src/main.rs:4415. The other one, in fast_start.rs:130, is local to the snapshot download.

So SIGTERM gets its default disposition for the whole replay, which can mean either of two things:

  • satd is PID 1 of its PID namespace, with no init in front of it. The kernel drops default-action signals sent to a namespace's init, so SIGTERM is ignored until the supervisor's SIGKILL. That matches the 600 s docker stop. The published image puts tini in front of satd (Dockerfile:240), so this depends on how the container was started; a package that execs satd as its entrypoint gets this.
  • Anywhere else. SIGTERM kills the process at once, mid-batch, the same as SIGKILL.

Neither is a graceful stop. Either way the replay ends unclean and #906 then throws it away.

During an ordinary sync

The handler exists by then, and Start9 still saw the stop run out its timeout on a StartOS box. Not yet explained. It needs a measurement of how long shutdown takes after SIGTERM during IBD, and what it waits on: the connect thread finishing its batch, the final flush of a large dbcache, or something else.

Expected

  • During a replay. SIGTERM (and SIGINT) stops satd at the next block boundary: it finishes the block it is connecting, flushes, leaves the rebuild marker in place, and exits 0. The time this takes must fit a supervisor's grace period. StartOS's sigtermTimeout and Docker's default 10 s are the reference points.
  • This holds whether or not satd is PID 1, so a package that runs satd without an init behaves the same as the published image.
  • During IBD. A stop after SIGTERM completes within a stated bound, and the log shows what shutdown is waiting on while it waits.

Acceptance

  • An integration test that sends SIGTERM partway through the connect phase of a -reindex and of a -reindex-chainstate. satd exits 0 within the bound, the marker is still there, and the next start continues the rebuild (with An interrupted full -reindex starts over instead of resuming from the block files #906).
  • The same with satd as PID 1 of a new PID namespace (for example unshare --pid --fork, or a container without --init).
  • SIGTERM during IBD of a regtest chain with large blocks: satd exits within the bound.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions