feat(dolt): drain the off-box backlog by CLI push before the sql-server starts (vp-6hb8) - #131
Merged
Merged
Conversation
…er starts (vp-6hb8) THE TRAP. Pushing over the sql-server pays a cold-open cost once per server lifetime per database: the git-blobstore transport has no server-side range reads, so the first push of a lifetime spools the store's whole remote blobset (measured: 822 MB / 311 git calls for a 1 GB store) while the listener's read_timeout — 15s in production, deliberately: it is the only working idle-connection reaper, wait_timeout being accepted but inert (verified empirically 2026-08-11, 8 idle connections alive at t+48s under wait_timeout=30) — kills the query mid-open. A store that misses one push accumulates backlog, which makes the next attempt larger, which makes it miss again: measured live, one store ratcheted 0 -> 8,514 unpushed commits in five days, unbackable the whole time. Every server restart re-arms the trap for every large store at once. Session-scoped timeouts are ignored by the listener; raising the global read_timeout reproduces the 2026-06-15 connection pileup; retry loops make literally zero progress (twelve consecutive attempts, backlog unchanged — the push dies during the remote-db open, before any chunk transmits). THE MECHANISM. Drain each database by CLI `dolt push` BEFORE the sql-server starts. startManagedDoltProcessWithOptions has just waited for the data-dir LOCK to be free, so no server owns the store and CLI access is safe — and a CLI push has no listener in front of it, so no read_timeout applies, no config raise/restore, no second restart. The server then boots cold but with ZERO backlog, and a cold push of a near-empty delta fits the production window (measured 2026-08-06: every store passed at 15s immediately after a drain, largest 13.2s), after which the store is warm and stays current via the patrol. FAILURE POSTURE. The drain never blocks the boot: a store with an unreachable or corrupted remote (five of nine fleet stores have hit remote-side "Blob not found" archive corruption) still needs its LOCAL server, so every per-database failure is loud and boot proceeds. It never force-pushes — a diverged store needs an ownership decision this code cannot make (vp-ukvx). The branch pushed is the database's OWN checked-out head from repo_state.json; an unreadable state is UNKNOWN and skipped with a reason, never pushed at a guessed branch. Budget exhaustion NAMES the databases it did not attempt (a truncated sweep that reads as complete is the vp-g2m4 shape). Default ON — the drain exists so an UNATTENDED restart cannot silently re-arm the ratchet — with GC_DOLT_BOOT_DRAIN=off for maintenance windows that manage draining themselves, and GC_DOLT_BOOT_DRAIN_BUDGET for the wall-clock cap (default 10m; a full-fleet drain after five days of backlog measured ~6m worst case). Fprintf returns to the operator log are explicitly ignored: best-effort logging must never block a boot. Mutations, each verified applied (anchor exactly once, mutant compiles) before its result counted; two initial mutants were themselves invalid Go (unused variable) or mis-anchored and were REDONE rather than recorded: M1 failure stops the sweep -> 1 failure M2 pushed at a guessed branch -> 1 failure M3 exhaustion stops naming the rest -> 1 failure M4 mangled repo_state pushed anyway -> 2 failures M5 kill switch defaults off -> 1 failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The trap this closes
Pushing over the sql-server pays a cold-open cost once per server lifetime per database: the git-blobstore transport has no server-side range reads, so the first push of a lifetime spools the store's whole remote blobset (measured: 822 MB / 311 git calls for a 1 GB store) while the listener's 15s
read_timeoutkills the query mid-open. A store that misses one push accumulates backlog, making the next attempt larger — measured live: 0 → 8,514 unpushed commits in five days, unbackable the whole time. Every server restart re-arms the trap.Every alternative was ruled out by measurement: session-scoped timeouts are ignored by the listener; raising the global read_timeout re-arms the 2026-06-15 connection pileup (
wait_timeoutverified inert — 8 idle connections alive at t+48s underwait_timeout=30); retry loops make literally zero progress (12 consecutive attempts, backlog unchanged — the push dies during the remote-db open, before any chunk transmits).The mechanism
Drain each database by CLI
dolt pushbefore the sql-server starts. The start path has just waited for the data-dir LOCK to be free, so no server owns the store and CLI access is safe — and a CLI push has no listener in front of it: no timeout raise, no restore, no second restart. The server boots cold but with zero backlog, which fits the production window (measured 2026-08-06: all nine stores passed at 15s immediately after a drain, largest 13.2s); the patrol keeps it current from there.Failure posture
Blob not foundarchive corruption; a dead remote must not brick the local server. Every failure is loud, boot proceeds.repo_state.json; unreadable state is UNKNOWN and skipped with a reason, never pushed at a guessed branch.GC_DOLT_BOOT_DRAIN=offto disable;GC_DOLT_BOOT_DRAIN_BUDGET, default 10m) — an unattended restart must not silently re-arm the ratchet.Verification
5 behavioural mutations, each verified applied (anchor exactly once, mutant compiles) before its result counted; two initial mutants were invalid Go or mis-anchored and were redone rather than recorded. Failure-stops-sweep → 1 red; guessed-branch → 1; silent exhaustion → 1; mangled-state-pushed → 2; kill-switch-default-off → 1.
Both repo governance guards were triggered and satisfied deliberately: the two new env vars are registered in the GC_* vocabulary golden, and the resource census's no-new-sleeps invariant pushed the budget test onto an injected clock seam (
bootDrainNowFn) — deterministic instead of timing-dependent.Deploy note
Takes effect on the next
gcbinary rollout + city restart; on that first boot it will also self-heal whatever backlog has accumulated by then.