Zero-downtime deploys: serve reads during WAL replay so single-writer handoff has no read/write gap

[tonyalaribe]

Summary

Even after the shutdown-bound fix (#82) and flush-throughput fix (#83), TF redeploys will still blip for old-drain + new-recovery (target ~30s) because the WAL is single-writer — only one process can own it at a time, enforced by the exclusive wal.lock flock (added in 92ad0b8). Achieving genuinely no-noticeable-downtime deploys requires a design change so that reads stay available during the handoff.

Background

The flock correctly serializes writers (prevents the concurrent-writer WAL fork / silent data loss). But serialization means during a deploy the incoming instance must wait for the outgoing one to release the lock, then replay the WAL before it can serve. During that window the pgwire port answers 57P03 ("starting up") — writes retry into PG+DLQ (recoverable), but reads (dashboards/queries) are unavailable.

Options to scope

• Serve reads from Delta+S3 while replaying the WAL — the incoming instance answers queries from the durable Delta tables (and optionally proxies recent rows from the outgoing instance) and only cuts over the write path once caught up.
• Split read vs write availability so query serving never depends on the writer's recovery.
• Read replica / shared queryable snapshot that stays up across writer restarts.

Prerequisite

Fix #82 (fast, bounded shutdown) and #83 (small WAL → fast replay) first — they shrink the handoff from ~10-15 min to ~30s, at which point zero-downtime becomes a polish rather than a necessity.

Severity / impact

Medium (product). Data is safe across deploys today (no silent loss; DLQ-recoverable). This issue is about eliminating the remaining read/write availability gap so deploys are invisible to users.

[tonyalaribe]

Blocked on / prereq: #82 (bounded shutdown) and #83 (flush throughput) — both shrink the handoff from ~10-15 min to ~30s before zero-downtime becomes worthwhile.