feat(cutover): stage pipeline + milestone highlights on the operator dashboard

[islandbitcoin]

Operator monitoring upgrades for the USDT cutover rehearsal (ENG-461) and cutover day.

What's new

• Pipeline view: stacked progress bar + clickable stage chips grouping the 14-status migration machine into 10 operator stages; chips filter the account table. Run context line shows state · runId · elapsed · % done.
• Highlights feed: server-derived milestone events — run started/COMPLETE/ROLLED BACK, provisioning complete, first account in flight, first fully migrated, 25/50/75/100% (with reconcile reminder), attention spikes/cleared, rollback started/drained. Bounded feed, every event also pino-logged so kubectl logs keeps history; a 30s background poll keeps milestones flowing with no browser open.
• Adaptive refresh: 15s while a run/rollback is active, 120s idle.
• New /api/milestones; /api/snapshot now carries stages + milestones.

Design

Stage grouping + milestone derivation are pure functions in operator-dashboard.ts (9 new unit tests). The script only wires and renders — already validated live on TEST via the ad-hoc dashboard pod (config-map override): 315 accounts, {pending: 315, pct 0, state pre, missingUsdt 315}, milestones flowing.

🤖 Generated with Claude Code

[islandbitcoin]

Deep review: 8 finder angles + adversarial verification of every candidate against the branch. The pure-function stage/milestone design with unit tests is solid, and the batched balance fetch legitimately fixes the 431 header-limit failure. Inline comments below, most severe first — the headline is the 100%-milestone denominator bug.

One pre-existing issue this PR amplifies (no diff line to anchor on): load() assigns the response body to the global snapshot before the response.ok check, so a transient 500 replaces the snapshot with {error} and subsequent renders throw on snapshot.accounts until the next successful poll — 8× more likely now with the 15s active-run cadence, and the new stage chips add render paths that hit it. Fix: const body = await response.json(); if (!response.ok) throw new Error(body.error || "Snapshot failed"); snapshot = body.

[islandbitcoin]

The “100% migrated — run reconciliation” milestone can never fire on real data (high)

This feeds all dashboard accounts into the stage summary, but migration records are only created for legacy_default discoveries — accounts classified already_usdt / residual_legacy_usd / missing_* keep migrationStatus: "none" forever. They count toward total (stage pending) but can never reach done (complete + skipped_already_migrated).

Your own TEST numbers show it: 315 manifest accounts vs 297 migrations ⇒ percentComplete caps at floor(297/315×100) = 94, so done === total never holds and the operator's cue to run reconciliation and hold per runbook (plus potentially the 75% threshold) never fires. The unit tests only use all-migratable status sets, which hides this.

Suggested fix: once the run has started, derive total from accounts that have (or will get) migration records — e.g. exclude permanently-none accounts using the preflight/discovery classification, or pass an eligibleTotal into summarizeCashWalletCutoverStages — keeping none visible as a pending chip only before prepare.

[islandbitcoin]

The elapsed-run timer is dead code — startedAt is never serialized (medium)

buildCashWalletCutoverOperatorSnapshot returns cutover: { state, cutoverVersion, runId, updatedAt } (operator-dashboard.ts:909–916) — no startedAt, and the CashWalletCutoverOperatorSnapshot type doesn't declare it either. The data exists (lifecycle.ts persists startedAt on start; it's in the mongoose schema); it just isn't plumbed through, so this condition is always false and “· elapsed …” silently never renders.

Fix server-side in operator-dashboard.ts:

cutover: {
  state: config.state,
  cutoverVersion: config.cutoverVersion,
  runId: config.runId,
  startedAt: config.startedAt?.toISOString(),
  updatedAt: config.updatedAt?.toISOString(),
},

plus startedAt?: string on the snapshot type’s cutover field.

[islandbitcoin]

Unguarded overlapping hydration passes during active runs (medium)

load() fires hydrateBalances un-awaited, and the 15s active-run reload starts a fresh multi-batch pass while the previous one (plus its 2s poll chain) is still running — balancePollTimer only guards the poll chain, not load()-initiated passes. Each /api/balances call also awaits snapshot(false) server-side with a 5s TTL, so a pass can stretch past 15s and passes then stack: duplicated request volume, status-line flicker between generations, interleaved renders.

Suggested change

	const BALANCE_BATCH = 60
	let balancePollTimer = null
	async function hydrateBalances(force) {
	if (!snapshot) return
	const walletIds = allWalletIds()
	if (walletIds.length === 0) return
	const url = "/api/balances?walletIds=" + encodeURIComponent(walletIds.join(",")) + (force ? "&refresh=1" : "")
	const response = await fetch(url)
	const result = await response.json()
	if (!response.ok) throw new Error(result.error || "Balance refresh failed")
	let fresh = 0, errors = 0, loading = 0, pending = 0, active = false
	mergeBalances(result.balances)
	renderSummary()
	renderRows()
	for (let i = 0; i < walletIds.length; i += BALANCE_BATCH) {
	const batch = walletIds.slice(i, i + BALANCE_BATCH)
	const BALANCE_BATCH = 60
	let balancePollTimer = null
	let hydrateGeneration = 0
	async function hydrateBalances(force) {
	if (!snapshot) return
	const generation = ++hydrateGeneration
	const walletIds = allWalletIds()
	if (walletIds.length === 0) return
	let fresh = 0, errors = 0, loading = 0, pending = 0, active = false
	for (let i = 0; i < walletIds.length; i += BALANCE_BATCH) {
	if (generation !== hydrateGeneration) return // superseded by a newer pass
	const batch = walletIds.slice(i, i + BALANCE_BATCH)

Also worth doing: on non-force poll passes, request only ids whose merged status is still loading instead of re-fetching all 600+ every 2s while the queue drains.

[islandbitcoin]

The 30s background poll rebuilds the full snapshot forever, even when idle (medium)

With snapshot-ttl-ms defaulting to 5000, every 30s tick misses the cache and re-runs buildSnapshot(): a discovery cursor over every unlocked account (one listByAccountId each) plus ~3–4 mongo queries per dashboard account — roughly 1,200+ sequential queries per tick for the TEST fleet — 24/7, including weeks after the run completes or rolls back. Pre-PR, builds only happened while a browser was polling.

Suggested change

	// Keep stage/milestone tracking alive even with no browser tab open.
	setInterval(() => {
	snapshot(false).catch((error) =>
	baseLogger.warn({ error }, "cutover dashboard background poll failed"),
	)
	}, 30_000)
	// Keep stage/milestone tracking alive even with no browser tab open.
	// Tight cadence only while a run/rollback is live; idle polls back off —
	// each rebuild is a discovery scan plus several mongo queries per account.
	const BACKGROUND_POLL_ACTIVE_MS = 30_000
	const BACKGROUND_POLL_IDLE_MS = 5 * 60_000
	const backgroundPoll = () => {
	snapshot(false)
	.catch((error) =>
	baseLogger.warn({ error }, "cutover dashboard background poll failed"),
	)
	.finally(() => {
	const active =
	lastStageSummary !== undefined &&
	(lastStageSummary.cutoverState === "in_progress" ||
	lastStageSummary.inFlight > 0 ||
	lastStageSummary.counts.rollingBack > 0)
	setTimeout(
	backgroundPoll,
	active ? BACKGROUND_POLL_ACTIVE_MS : BACKGROUND_POLL_IDLE_MS,
	)
	})
	}
	setTimeout(backgroundPoll, BACKGROUND_POLL_ACTIVE_MS)

[islandbitcoin]

Client-side STAGES hand-copies CASH_WALLET_CUTOVER_STAGE_STATUSES (cleanup — flagged independently by three review angles)

The HTML is a template literal in a module that already imports from operator-dashboard — the status lists can be interpolated once instead of hand-maintained in two files. As-is, the next status added to the state machine (#433 touched transitions the same week) updates server counts but not this copy: the chip filter silently hides accounts in the new status while the chip still shows a nonzero count.

Suggested change

	const STAGES = [
	["pending", "Pending", "#98a2b3", ["none", "not_started"]],
	["provisioning", "Provisioning", "#7a5af8", ["started", "provisioned"]],
	["moving", "Moving funds", "#1f5eff", ["balance_read", "invoice_created", "balance_move_sending", "balance_move_sent", "balance_move_verified"]],
	["fees", "Fees", "#0ba5ec", ["fee_reimbursement_invoice_created", "fee_reimbursement_sending", "fee_reimbursed"]],
	["finalizing", "Finalizing", "#12b76a", ["pointer_flipped", "legacy_zero_verified"]],
	["complete", "Complete", "#16794c", ["complete"]],
	["skipped", "Skipped (USDT)", "#66c61c", ["skipped_already_migrated"]],
	["attention", "Attention", "#b42318", ["failed", "requires_operator_review"]],
	["rollingBack", "Rolling back", "#946200", ["rollback_started"]],
	["rolledBack", "Rolled back", "#667085", ["rolled_back"]],
	]
	const stageStatuses = (key) => (STAGES.find((s) => s[0] === key) || [null, null, null, []])[3]
	const STAGE_STATUSES = ${JSON.stringify(CASH_WALLET_CUTOVER_STAGE_STATUSES)}
	const STAGES = [
	["pending", "Pending", "#98a2b3"],
	["provisioning", "Provisioning", "#7a5af8"],
	["moving", "Moving funds", "#1f5eff"],
	["fees", "Fees", "#0ba5ec"],
	["finalizing", "Finalizing", "#12b76a"],
	["complete", "Complete", "#16794c"],
	["skipped", "Skipped (USDT)", "#66c61c"],
	["attention", "Attention", "#b42318"],
	["rollingBack", "Rolling back", "#946200"],
	["rolledBack", "Rolled back", "#667085"],
	].map(([key, label, color]) => [key, label, color, STAGE_STATUSES[key]])
	const stageStatuses = (key) => STAGE_STATUSES[key] || []

(Add CASH_WALLET_CUTOVER_STAGE_STATUSES to the import from @app/cash-wallet-cutover/operator-dashboard.)

[islandbitcoin]

Stage chip + migration dropdown can contradict → silently empty table (low)

Chip clicks clear the dropdown, but a dropdown change doesn't clear stageFilter (the generic change handler only calls renderRows). Both are ANDed here, so picking a status outside the active chip's stage renders zero rows while the chip still shows a nonzero count — misleading during an incident.

Suggested change

	if (f.migration && account.migrationStatus !== f.migration) return false
	if (stageFilter && stageStatuses(stageFilter).indexOf(account.migrationStatus) === -1) return false
	// An explicit status pick overrides a stale stage chip (chip clicks
	// clear the dropdown, but not vice versa).
	if (f.migration) return account.migrationStatus === f.migration
	if (stageFilter && stageStatuses(stageFilter).indexOf(account.migrationStatus) === -1) return false

(Alternative: reset stageFilter + renderPipeline() in the dropdown's change handler.)

[islandbitcoin]

Milestone text lands in innerHTML unescaped (low)

m.text interpolates runId (free-form CLI string, --run-id), and this is the one unescaped sink — the pipeline note correctly uses textContent. Effectively self-XSS on an internal ops dashboard, but cheap to close:

Suggested change

	feed.innerHTML = items.map((m) =>
	'<li><span class="dot ' + m.kind + '"></span><span class="t">' +
	new Date(m.at).toLocaleTimeString() + '</span><span>' + m.text + '</span></li>'
	).join("") || '<li><span class="muted">no events yet</span></li>'
	const esc = (s) => String(s).replace(/[&<>"']/g, (c) =>
	({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[c]))
	feed.innerHTML = items.map((m) =>
	'<li><span class="dot ' + esc(m.kind) + '"></span><span class="t">' +
	new Date(m.at).toLocaleTimeString() + '</span><span>' + esc(m.text) + '</span></li>'
	).join("") || '<li><span class="muted">no events yet</span></li>'

[islandbitcoin]

Self-chaining reload dies permanently if one load() never settles (low)

The next reload is only armed when the previous load() settles. Server-side, all snapshot callers serialize behind one un-timed-out pending promise — a hung DB call leaves /api/snapshot unresponsive, and the Refresh button doesn't re-arm the chain either. The old setInterval recovered by construction; a watchdog restores that guarantee:

Suggested change

	function scheduleReload() {
	if (reloadTimer) clearTimeout(reloadTimer)
	reloadTimer = setTimeout(() => {
	load(false)
	.catch((error) => { $("status").textContent = error.message })
	.finally(scheduleReload)
	}, isActiveRun() ? 15000 : 120000)
	}
	function scheduleReload() {
	if (reloadTimer) clearTimeout(reloadTimer)
	reloadTimer = setTimeout(() => {
	// Watchdog: re-arm even if load() never settles (the server can hang
	// behind a stuck snapshot build with no timeout).
	const watchdog = setTimeout(scheduleReload, 60000)
	load(false)
	.catch((error) => { $("status").textContent = error.message })
	.finally(() => { clearTimeout(watchdog); scheduleReload() })
	}, isActiveRun() ? 15000 : 120000)
	}

[islandbitcoin]

All findings addressed in e72edce4. Point-by-point:

• 100% milestone denominator (the headline) — new eligibleTotal on the stage summary: once the run has started, permanently-none accounts (already_usdt / residual / missing_*) are excluded from the denominator; before start everything counts, so the pre-run pending view is unchanged. percentComplete, the 25/50/75 thresholds, and the 100%-reconciliation cue all use it. Tests updated with mixed-none fixtures (including the exact 315-vs-297 shape that hid this), and the old all-migratable 100% test was corrected — under the new semantics its previous state was already fully done.
• Elapsed timer dead code — startedAt plumbed through the snapshot cutover block + type; the client condition now actually fires.
• Overlapping hydration — generation counter guards load()-initiated passes; the 2s poll chain now requests only still-loading wallets instead of all 600 per pass.
• Idle background poll — 30s only while a run/rollback is live, 5m when idle (self-rescheduling, decided per-tick from the last stage summary).
• STAGES hand-copy — client stage lists interpolated from CASH_WALLET_CUTOVER_STAGE_STATUSES at render time; one source of truth.
• Chip/dropdown contradiction — explicit status pick overrides the stage chip.
• Milestone innerHTML — escaped (kind + text).
• Reload watchdog — re-arms after 60s even if a load() never settles.
• Pre-existing load() bug (review body) — body parsed and ok-checked before assigning the global snapshot.

9 suites / 55 tests green.