chain_plugin: first-class chain-sync gate — is_synced() + synced() signal; gate operator-daemon startup

[jglanz]

Problem

Every test cluster's boot window logged 3–11 FATAL-class lines per run:

error  batch_operator: table read threw sysio.chains::chains -- Account Query Exception (3060002)
error  batch_operator: table read threw sysio.chains::chains -- Contract Table Query Exception (3060003)

Mechanism (forensically verified on preserved clusters): operator daemon nodes start ~2 min after the depot contracts deploy — the harness ordering is already correct — but each daemon node is a fresh nodeop replaying from genesis, and batch_operator_plugin::plugin_startup eagerly read sysio.chains ~0.7 s post-spawn, before the node's local replay reached the deploy blocks. One-shot per daemon, self-resolving at the first synced tick (+13–14 s), but every boot paid the error-log noise.

The underwriter_plugin already solved this for itself with a private sync gate (defer startup until the last-irreversible block time is within 5 s of wall-clock, observed via accepted_block).

Change

Promote that machinery to a first-class chain_plugin API, consumed by both operator-daemon plugins:

• chain_plugin::is_synced(recency = 5s) — LIB-recency predicate (fork_db_root().block_time() within recency of now). LIB, not head, deliberately: operator daemons run read-mode = irreversible, so this is the state their table reads actually serve.
• chain_plugin::synced() → boost::signals2::signal<void()>& — the codebase-standard signal shape (accessor returning the impl-owned signal, like the controller's accepted_block() and batch_operator_plugin::debugging_opp_envelope()). Contract:
  • Slots run on the main thread from the application executor queue, after the triggering block fully commits — never mid block-application, where table reads observe an incomplete view (the underwriter's hard-won bug, now encoded once in the emitter instead of in every consumer).
  • Emitted on each stale→synced transition (edge detector, is_replaying()-guarded), not once per process — a once-per-process latch that fires with no consumer connected while the chain then stalls would leave a later subscriber waiting forever. One-shot consumers disconnect inside their slot.
• sync_gate.hpp (new, sysio::chain_apis) — the pure building blocks: default_sync_recency_ms, block_time_is_recent (moved verbatim from the underwriter's sync_detail.hpp), lib_time_is_recent, and the sync_transition edge detector. Header-only and unit-testable without appbase.

Consumers (identical two-branch idiom — is_synced() fast path inline, else connect):

• underwriter_plugin sheds its private copy (net-negative diff): sync_recency_ms, block_time_is_recent, chain_is_synced(), and the per-block re-check + inner executor post all deleted; the gate connection now targets chain_plug->synced(). Behavior unchanged.
• batch_operator_plugin gains the gate: the startup body (outpost discovery → private cron_service sizing/creation → epoch_tick → relay jobs) moved verbatim into impl::run_deferred_startup(), run inline when already synced (escape aborts node startup, as today) or from the synced() slot via a contained wrapper (an escape from a posted task would tear down the whole node). Bonus: cron thread sizing now sees the real outpost count instead of always hitting the MIN_CRON_THREADS floor on cold boots.

No harness (wire-tools-ts) changes — the phase ordering was already correct.

Tests

• New plugins/chain_plugin/test/test_sync_gate.cpp (sync_gate_tests): the 7 recency-predicate cases migrated from test_underwriter_sync.cpp, 4 pure sync_transition edge cases (initial-stale, fires-once, no-refire, re-arm), and 2 tester-backed lib_time_is_recent cases (stale genesis chain → false; block time jumped to now + finalized → true).
• test_underwriter_sync.cpp keeps its gate-payload cases; the migrated predicate cases were removed.
• batch_operator intentionally gets no new unit tests: the change is a verbatim code-move plus glue over the tested signal contract; gate behavior is covered by the chain_plugin tests and the live validation below.

Verification

Unit sweep (all green):

• test_chain_plugin (incl. sync_gate_tests — 13 cases), test_underwriter_plugin, test_batch_operator_plugin, plugin_test, unit_test -- --sys-vm (1507/1507), test_fc

Reference-data regeneration (bundled). deep_mind_tests and savanna_misc_tests/verify_block_compatibitity were already failing on pristine master at this local build (action-merkle / block-id drift between the committed reference data and the current contract build — reproduced byte-identically with this change stashed out, so unrelated to the plugin change). Regenerated per wire-sysio/CLAUDE.md (--save-dmlog, --save-blockchain): unittests/deep-mind/deep-mind.log + unittests/test-data/consensus_blockchain/{blocks.log,id,snapshot}. With them, unit_test is 1507/1507.

Live validation — full 13-flow suite against the rebuilt nodeop (fresh cluster per flow, canonical runner + heartbeat monitor, 4 concurrent): 13/13 SUCCEEDED (flow-operator-collateral-deposit, flow-batch-operator-termination, flow-batch-operator-slashing, flow-swap-{with-underwriting,to-wire,from-wire,variance-revert,non-native-tokens,private-reserves}, flow-reserve-lifecycle, flow-node-owner-nft, flow-emissions-soak, flow-yield-distribution). Every one: boot-window 3060002/3060003 count = 0 (baselines 3–11), every operator daemon armed (waiting for chain sync) then released (chain synced — starting deferred startup).

Boot-window assertions:

• flow-operator-collateral-deposit (baseline fatal=4) and flow-batch-operator-termination (9-op, baseline fatal=9): heartbeat fatal=0 through the boot window; each batch daemon logs waiting for chain sync before outpost discovery → chain synced — starting deferred startup; one chain_plugin node is synced ilog per node.
• flow-swap-with-underwriting: underwriter behavior unchanged (its slot now runs via chain_plugin's posted emission instead of its own inner post — cosmetic log-ordering delta only).
• All other flows: fatal=0 through the boot window (the 3060002/3060003 class is zeroed cluster-wide).

Risks

• Shutdown ordering — posted emissions are drained-without-executing on quit (appbase contract); consumer slots guard shutting_down; connections reset in each consumer's plugin_shutdown, which runs before chain_plugin's.
• Late subscription — covered by the is_synced() fast path plus re-arming edge semantics (no dead-latch case).
• Snapshot-started nodes — correctly not-synced until caught up; the gate defers daemon startup exactly as intended.
• Fresh-genesis producers — initial_timestamp ≈ now → inline fast path, zero added latency.
• Steady-state cost — one fork_db root read per accepted block (same probe the underwriter's private gate already paid while gated, now shared and is_replaying()-guarded).

Pre-existing signatures NOT addressed here (out of scope)

Two error-level log signatures appear in some flow runs; both are pre-existing and independent of this change (all 13 flows still exit 0):

1. msgch::bootstrap → 3090004 missing authority of sysio.msgch. batch_operator_plugin::do_poll_epoch_state pushes msgch::bootstrap at epoch 0 signed with the operator's active authority, but the action's require_auth(get_self()) requires sysio.msgch's own authority — so the push fails unconditionally at the auth check (never reaching check(epoch == 0)), regardless of timing. The initial advance is driven by the system/harness (which holds sysio.msgch's keys), so it is benign; this gate does not change it. A follow-up removes the operator-side push (the advance is a system responsibility under the contract's authority model).
2. MockYieldEmitter: externalEpochRef not monotonic. A single ETH-side revert in flow-yield-distribution's emission scenario — a wire-ethereum contract path, unrelated to depot plugin startup, non-fatal to the flow.

🤖 Generated with Claude Code

[heifner]

Superseded by #522 — same goal, reduced mechanism, incorporating the review discussion: the predicate lands as controller::is_synced() beside the fork_db accessors it reads, and the daemons gate on the existing irreversible_block channel instead of a new signal (channel deliveries are already posted post-commit, which was the signal's load-bearing contract). #522 also folds in the follow-ups from review: single gated path (no already-synced fast path — the daemons can never be synced at plugin_startup in our topology), uniform fail-fast on terminal deferred-startup failure, required read-mode = irreversible, and no reference-data regen (it is based past #519, which is what this branch conflicts with). The one-file follow-up stacked here (#521) is restacked onto #522.