walrus sync-standby: durable synchronous-standby WAL receiver

[achudnovskij]

What

Adds walrus sync-standby <dir> — the receiver half of Ubicloud sync_pair HA: it streams WAL from a primary as a durable synchronous standby, fsyncs WAL to local disk, and acks the flushed LSN to join the primary's synchronous_standby_names = ANY 1 quorum alongside the streaming standby. Self-contained under src/pg/sync_replica/ with its own CLI subcommand, decoupled from the Uploader receiver (wal::receive).

Draft — opening for review; end-to-end re-validation on the rebased code is in progress (see Testing).

Architecture (two threads, bridged by Shared)

• receive.rs — the synchronous (tokio-free) hot path: blocking socket I/O + positioned write_all_at + fdatasync, so a sole-acker commit pays no task-scheduler latency. Durability invariant: the flush ack never precedes the fdatasync of the bytes it covers.
• controller.rs — its own tokio runtime: the sole-acker poller, the retention janitor + back-pressure, and the mTLS control API.
• shared.rs — lock-free atomics (fsyncd_lsn, sole_acker, ack_ceiling) + a Notify.

Diagrams + lifecycles in src/pg/sync_replica/docs/ (architecture, state machines, WAL record + segment lifecycles, side-channel queries).

Capabilities

• mTLS control API (api.rs): GET /v1/status, POST /v1/dr-catchup, POST /v1/failover-primary — the endpoints the Ubicloud CP drives on failover.
• Sole-acker detection: the poller reads the peer standby's flush_lsn from pg_stat_replication; when the standby is the pacing acker (2-acker) the receiver coalesces fsyncs behind a ~1 ms batch window (~4× lower IOPS at no commit-latency cost); when it's the only acker it fsyncs per-frame for minimum latency.
• dr-catchup (dr_tail.rs): upload the retained receiver-only WAL tail to a DR-tail S3 lane so a promotion candidate can replay it (contiguous-durable gate + from-anchor → RPO=0 across standby-behind / both-down / total-loss).
• Retention janitor + back-pressure; reconnect/retarget loop (a stream break or failover-primary ends the session, not the process — the control API stays up and the receiver resumes from its durable frontier).

Deployment

docker/wal-receive/entrypoint.sh execs walrus sync-standby (the operator uses the image ENTRYPOINT, no operator command change). Requires the primary to grant the replication role pg_read_all_stats so the sole-acker poller can read flush_lsn (done in the Ubicloud repo: post-installation-script + the receiver nexus).

Review notes (rebase reconciliation)

This branch was squashed to one commit and rebased on main, which independently reworked config (Vars + json) + TLS (TlsParams) + the handle signature. Reconciliation choices:

• wal/receive.rs is taken verbatim from main (the Uploader is untouched by this feature).
• The sync-standby path was adapted to main's Vars-based config: the CLI resolves PgConfig::resolve(&vars) + slot_name(&vars) and passes them to sync_replica::run (mirrors wal-receive); build_dr_s3_storage uses s3_config(&Vars::default(), …).
• maybe_upgrade_sync now threads TlsParams (matches main's async maybe_upgrade).
• Minor redundancy to note: query_wal_segment_size exists both in conn.rs (for sync_replica) and privately in wal/receive.rs (main's Uploader) — a candidate follow-up to unify.

Testing

• cargo fmt / cargo clippy --all-targets -D warnings / cargo test — 497 lib tests pass, clippy clean, on the rebased commit.
• Pre-rebase, all 6 sync_pair failover edge cases passed at RPO=0 (recv-down, standby-behind, write-block, both-down, budget-backpressure, total-loss). Re-running the E2E suite against the rebased image is in progress.

🤖 Generated with Claude Code

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}