Chain: gate operator-daemon startup on controller::is_synced()

[heifner]

Replaces #520 — same goal, reduced mechanism.

Problem

Every test cluster's boot window logged error-class lines from batch_operator (Account/Contract Query Exception 3060002/3060003): plugin_startup eagerly read sysio.chains before a cold-booting operator node's local replay reached the deploy blocks. The underwriter already solved this privately with an accepted_block-signal gate plus an inner executor re-post.

Change

• controller::is_synced(now, recency_window) plus a wall-clock is_synced(): the LIB-recency predicate (fork_db root block time within controller::default_sync_recency_ms of now), computed on demand, never cached. Lives beside the fork_db accessors it reads and is thread-safe like them. LIB rather than head deliberately: operator daemons run read-mode = irreversible, so the irreversible state is what their table reads serve.
• No new signal. The predicate is a level, and the wake-up a gated consumer needs already exists as the channels::irreversible_block appbase channel — a LIB advance is the only event that can turn the predicate true, and channel deliveries are posted to the application executor, so callbacks run on the main thread after the triggering block fully commits (the post-commit contract the underwriter previously re-implemented with its inner post). Steady-state cost is zero: the daemons are subscribed only while gated, and channel::publish skips posting with no subscribers.
• underwriter_plugin: the gate subscription moves from the raw controller accepted_block signal to the channel; the private predicate, sync_recency_ms, and the inner re-post are deleted. Single gated path — there is deliberately no already-synced fast path: operator daemons boot with genesis-stale LIB in every topology, and a node that somehow is synced at startup is released by the next LIB advance.
• batch_operator_plugin: the startup body (outpost discovery, cron_service sizing/creation, epoch_tick, relay jobs) moves verbatim into impl::run_deferred_startup behind the same gate — the boot-window read errors disappear, and cron sizing sees the real outpost count instead of always hitting the MIN_CRON_THREADS floor.
• Uniform fail-fast: a terminal deferred-startup failure logs and quits the node in both daemons — an operator daemon that is not performing its duties is supervisor-visible instead of hidden behind a running process. The underwriter's bounded preflight retry grace still absorbs transients; the escape_policy split (abort vs contain, previously selected implicitly by restart timing) is deleted, with underwriter_detail::is_terminal_failure classifying the terminal states in one place.
• read-mode = irreversible is required by both daemons (FC_ASSERT at plugin_initialize when the plugin is enabled). Together with producer_plugin's inverse assert (no producer-name under irreversible read-mode), producer/daemon co-hosting is now impossible by configuration.

Behavior deltas vs #520

• No already-synced fast path; release comes at the next LIB advance (in practice the daemons are never synced at plugin_startup).
• Terminal deferred-startup failure quits the node. Previously a terminally failed underwriter stayed up serving preflight_failed/wiring_failed gate payloads over /v1/underwriter/* indefinitely; that surface now exists during the retry grace and the shutdown window only. Monitoring that polls a failed-but-alive underwriter will see process exit instead.
• No reference-data changes ride along (this branch is based past Chain: Remove base_per_transaction_net_usage consensus parameter #519, so the regenerated-data conflicts on chain_plugin: first-class chain-sync gate — is_synced() + synced() signal; gate operator-daemon startup #520 do not carry over).

Considered and deliberately not replaced

Other in-sync checks in the tree measure different things and stay as they are: producer_plugin's max-irreversible-block-age production gate (needs the raw age for its log/API surface and deliberately treats a snapshot-started producer as synced so it can revive a chain), block_state::is_recent's 30s head-recency vote gate (gating votes on LIB recency would deadlock finality recovery — votes are what create finality), producer_plugin's stale-production wake (head-based so producers can revive a stale chain), net_plugin's peer-relative sync_manager, and the vote-recency production pause.

Tests

unittests/controller_sync_tests.cpp: the boundary cases drive is_synced(now, window) with instants offset from the tester chain's actual last-irreversible block time (window edge, past-window, zero window, clock-skew-ahead, replay gap), so every comparison exercises the real controller path deterministically; wall-clock cases cover the convenience overload. The underwriter's migrated predicate cases were removed with its private predicate; its gate-payload cases remain, plus a pinning test for is_terminal_failure.

[huangminghuang]

Re-reviewed at f17feb8. The sync-gate restart race is a narrow, self-recovering operational edge case; the documented tradeoff is acceptable and non-blocking. No blocking findings.