fix(cutover): harden migration + rollback from ENG-461 rehearsal findings (ENG-401/483/484)

[islandbitcoin]

Summary

Hardening from the ENG-461 TEST cutover (real-money rehearsal → actual cutover, 297/297 complete). Every fix was hit live and root-caused on TEST; the rollback fix is validated against real IBEX there.

1. Rollback reverse payment overspend → IBEX 400 (ENG-401)

The reverse step paid back destinationAmountUsdtMicros (the forward target), but the USDT wallet legitimately holds slightly less: the un-reimbursed forward routing fee plus the balance read rounding up (e.g. real 0.443691959514 → read 443692). Paying the target verbatim overspends → IBEX 400 → account stranded pointer-reverted with funds in USDT.

Fix: new floored readDestinationSpendableUsdtMicros; reverse min(spendable, target); fail closed only on a shortfall beyond the 1-cent tolerance (signature of genuine post-cutover spend). Validated on TEST: $0.52 account → rolled_back, funds returned to legacy USD, pointer restored.

2. IBEX 429s mass-fail balance reads (ENG-483)

Unthrottled run-batch mass-failed 233/297 accounts with FetchError: Too Many Requests — balance reads were the one IBEX path not wrapped in withIbexRateLimitRetry, and the wrapper only handled returned errors while some client paths throw FetchError. Now every account-details read retries, thrown 429s retry too, and run-batch defaults to --step-delay-ms 2000 (~30/min, the empirically safe rate; 0 still available explicitly).

3. Dust fees + retry-failed operator command (ENG-484)

• Sub-cent fee reimbursements (88–515 micros observed) are below IBEX's minimum payable amount → 400 at pay → 9 migrations stranded. Fees under one cent are now absorbed (skip to fee_reimbursed, absorbed micros recorded for audit).
• New retry-failed CLI command replaces manual mongo surgery: resets failed migrations to their last safe status, field-driven (pointer_flipped when the pointer moved, else not_started with stale progress cleared). Never touches records with a payment transaction. Supports --account-id / --dry-run.

4. Treasury resolves to the funder's USDT wallet (ENG-482)

getTreasuryWalletId returned the funder's default wallet; where that default is BTC, USDT fee reimbursements 404 at IBEX — which stranded every funded account in the rehearsal until the TEST funder's default was manually flipped. The cutover treasury resolver now finds the funder's USDT wallet regardless of default (mirroring the dealer resolver's find-by-currency pattern), with a descriptive error when the funder has no USDT wallet. The global funderWalletResolver is unchanged.

5. audit-accounts data-quality preflight (ENG-484)

Pointer flip saves the whole account document, so mongoose re-validates every field — accounts with pre-existing invalid data (empty username, null statusHistory) fail mid-migration. Hit twice live on TEST; the same artifact was also found on the dealer + bankowner system accounts. New audit-accounts command sweeps all account documents through validateSync() and reports what would fail — a runbook D1 prerequisite before prepare.

Test plan

• 63 unit tests pass (test/flash/unit/app/cash-wallet-cutover/), 14 new
• Fix 1 validated end-to-end on TEST against real IBEX (clean rolled_back, funds returned)
• Fix 3 (dust) mirrors the manual straggler handling that completed the TEST cutover
• Fix 4 short-circuit path is live TEST behavior (funder default = USDT)

🤖 Generated with Claude Code

[linear]

ENG-461

ENG-401

ENG-484

[islandbitcoin]

Deep review: 8 finder angles (line-by-line, removed-behavior, cross-file tracing, reuse, simplification, efficiency, altitude, conventions) + adversarial verification of every candidate against the branch. The core fixes verified correct — the spendable-floor rollback math, the thrown-429 retry wrapper, retry-failed's money-moved guards, and the $unset transition plumbing all check out. Inline comments below are the surviving findings.

Two follow-up notes that don't map to a diff line:

• 429 detection string-matches error messages because IbexError discards the structured .status (httpCode is commented out in src/services/ibex/errors.ts). Fixing httpCode propagation in the IBEX service layer would make detection === 429 for all IBEX callers — the bridge error mapper already does exactly this for its 429s.
• src/servers/exporter.ts:164 still gauges the funder default wallet. Since ENG-482's whole point is that the treasury is now a non-default USDT wallet, treasury drain during the cutover is invisible to the funder balance metric and its alerts.

[islandbitcoin]

Fee absorption threshold swallows fees the old code successfully reimbursed (medium)

On main, only feeAmountUsdtMicros === "0" was skipped — fees of 1–9999 micros were reimbursed via the no-amount invoice path, which exists precisely because fixed invoices under 10,000 micros can't be created (the pre-existing spec proves it paid 4735 micros). This alias absorbs everything under 10,000, but the rehearsal only observed IBEX 400s at 88–515 micros: the 516–9999 band is absorbed with no evidence it ever failed (~19× the observed failure band).

It also welds a customer-funds policy ("which fees do users eat") to an invoice-plumbing constant — tuning CUTOVER_MIN_FIXED_USDT_INVOICE_MICROS for invoice-format reasons would silently widen absorption — and it makes the no-amount branch in createCashWalletMigrationFeeReimbursementInvoice dead code for new runs.

Suggested change

	// ENG-484: IBEX rejects payments below its minimum payable amount, so
	// sub-cent fee reimbursements (88–515 micros observed in the ENG-461
	// rehearsal) 400 at the pay step and strand the migration. Fees under one
	// cent are absorbed instead of reimbursed — invisible to users.
	export const isSubMinimumFeeReimbursementAmount = (
	feeAmountUsdtMicros: string,
	): boolean | InvalidCashWalletCutoverAmountError =>
	usesNoAmountFeeReimbursementInvoice(feeAmountUsdtMicros)
	// ENG-484: IBEX rejects payments below its minimum payable amount, so
	// sub-minimum fee reimbursements (88–515 micros observed in the ENG-461
	// rehearsal) 400 at the pay step and strand the migration. Absorb only fees
	// below IBEX's minimum payable — NOT the fixed-invoice threshold above: the
	// no-amount invoice path exists precisely to pay 1–9999-micro amounts and
	// reimbursed 4735-micro fees successfully before ENG-484.
	const CUTOVER_MIN_PAYABLE_USDT_MICROS = 1_000n // TODO: confirm IBEX's real minimum
	export const isSubMinimumFeeReimbursementAmount = (
	feeAmountUsdtMicros: string,
	): boolean | InvalidCashWalletCutoverAmountError => {
	if (!/^\d+$/.test(feeAmountUsdtMicros)) {
	return new InvalidCashWalletCutoverAmountError(
	`Invalid non-negative integer amount: ${feeAmountUsdtMicros}`,
	)
	}
	return BigInt(feeAmountUsdtMicros) < CUTOVER_MIN_PAYABLE_USDT_MICROS
	}

(Set the constant to IBEX's actual minimum once confirmed, and update the worker.spec.ts expectations — "9999" should become payable again via the no-amount path.)

[islandbitcoin]

Absorbed fees will mass-flag false “shortfall” rows in the reconciliation audit (medium)

computeCutoverBalanceAudit (operator-dashboard.ts:504–552, untouched by this PR) compares the destination delta against destinationAmountUsdtMicros with zero tolerance and flags status: "shortfall" on any positive deficit. After the forward move the destination wallet holds exactly target − fee, and only the fee reimbursement (paid to the destination wallet) topped it back up.

With fees now absorbed, every completed migration with an absorbed fee (88–9999 micros — widespread in the rehearsal) renders as a red shortfall row during the post-run reconciliation hold — drowning real shortfalls exactly when the runbook says to look.

Suggest giving the audit the same sub-cent tolerance the rollback got, or subtracting recorded-but-unpaid feeAmountUsdtMicros (i.e. feeReimbursementPaymentTransactionId unset) from the expected minimum.

[islandbitcoin]

Rollback can attempt a 0-micro or sub-minimum-payable reverse payment (low)

There's no lower bound on reverseAmount. A 1-cent account (target = 10000) with a drained wallet gives shortfall = 10000, which is not > the 10,000 tolerance — the gate passes and payInvoice is called with "0". Even the normal 1-cent rollback (only the 88–515-micro forward fee missing) produces a reverse amount of ~9,500–9,900 micros — below the very minimum this PR documents in ENG-484 — so IBEX 400s and the migration lands in requires_operator_review with a misleading payment error instead of the old clear fail-closed message.

Suggested change

	const reverseAmount = spendable < target ? spendable : target
	const shortfall = target - reverseAmount
	if (shortfall > ROLLBACK_SHORTFALL_TOLERANCE_USDT_MICROS) {
	return new CashWalletMigrationFailedError(
	`USDT balance ${usdtBalance} no longer covers reverse amount ${current.destinationAmountUsdtMicros}`,
	`USDT balance ${spendableStr} is short of reverse target ${target} by ${shortfall} micros (> tolerance) — likely spent post-cutover`,
	)
	}
	const reverseAmountStr = reverseAmount.toString()
	const reverseAmount = spendable < target ? spendable : target
	const shortfall = target - reverseAmount
	if (shortfall > ROLLBACK_SHORTFALL_TOLERANCE_USDT_MICROS) {
	return new CashWalletMigrationFailedError(
	`USDT balance ${spendableStr} is short of reverse target ${target} by ${shortfall} micros (> tolerance) — likely spent post-cutover`,
	)
	}
	// IBEX rejects payments below its minimum payable amount (ENG-484): a
	// drained or 1-cent wallet can pass the tolerance gate with a 0- or
	// sub-minimum reverse amount — fail closed with a clear message instead
	// of a doomed pay attempt and a misleading IBEX 400.
	if (reverseAmount < ROLLBACK_MIN_PAYABLE_USDT_MICROS) {
	return new CashWalletMigrationFailedError(
	`Reverse amount ${reverseAmount} micros is below the minimum payable — target ${target}, spendable ${spendableStr}; resolve manually`,
	)
	}
	const reverseAmountStr = reverseAmount.toString()

(Define ROLLBACK_MIN_PAYABLE_USDT_MICROS next to ROLLBACK_SHORTFALL_TOLERANCE_USDT_MICROS at the top of this file.)

[islandbitcoin]

Boundary band where the treasury reimburses post-cutover user spending (low)

Step 3's legacy top-up uses the same 10,000-micro strict-greater threshold on sourceBalance − current. With user spend s, un-reimbursed forward fee f, and reverse-payment receive dust d: when f+s ≤ 10000 here but f+s+d > 10000 at step 3, the treasury pays the whole f+s+d — including the user's own spend (e.g. s=9600, f=300, d=150 → treasury pays 10,050 micros, 9,600 of it the user's own consumption). The old code failed closed on any shortfall, so this is a categorical change.

Exposure is bounded at ~1 cent + fee + dust per account, so it may be an acceptable tradeoff — but it deserves an explicit comment here, or subtract the step-2 shortfall (known spend) from the step-3 top-up.

[islandbitcoin]

This comment's justification is false for legacy_zero_verified (low — comment fix)

legacy_zero_verified is missing from the runner's AMBIGUOUS_SIDE_EFFECT_STATUSES, so a failure there — after the balance move AND pointer flip — routes to failed. Money-moved migrations do reach failed. The widening is still safe today because retry-failed's guards hold (verified: balanceMovePaymentTransactionId is persisted in the same transition that enters balance_move_sending, and pointer-flipped records previousDefaultWalletId → resume at pointer_flipped), but the comment shouldn't promise what the machine doesn't enforce:

Suggested change

	// Operator retry (ENG-484): `failed` only ever comes from non-ambiguous
	// statuses (the runner routes ambiguous ones to requires_operator_review),
	// so re-driving is safe. Two resume points: `not_started` re-walks the whole
	// pre-money path; `pointer_flipped` resumes a post-pointer failure without
	// touching previousDefaultWalletId.
	// Operator retry (ENG-484). NOTE: `failed` CAN hold money-moved migrations —
	// a failure during legacy_zero_verified (balance moved AND pointer flipped)
	// routes to `failed` because that status is not in the runner's
	// AMBIGUOUS_SIDE_EFFECT_STATUSES. These edges are only safe behind
	// retry-failed's guards: skip any migration with a recorded payment
	// transaction id, and resume at `pointer_flipped` (never `not_started`)
	// when previousDefaultWalletId is set.

Deeper fix worth considering: add legacy_zero_verified to AMBIGUOUS_SIDE_EFFECT_STATUSES in runner.ts so post-pointer failures go to operator review like the other money-moved statuses.

[islandbitcoin]

A mid-loop race aborts the run and discards the report of migrations already reset (low)

If a listed migration is concurrently moved out of failed (e.g. an operator rollback-request — failed is rollbackable), the guarded findOneAndUpdate matches nothing, transitionMigration returns CouldNotUpdateError, and this returns immediately — but earlier resets are already committed (each transition is an independent write). The operator's CLI shows only an error with no migrationIds/counts for what was already re-armed.

Suggested change

	if (transitioned instanceof Error) return transitioned
	if (transitioned instanceof Error) {
	// Lost a race (e.g. a concurrent rollback-request moved this migration
	// out of `failed`). Skip it and keep going — aborting would discard
	// the report of migrations already reset.
	report.eligible -= 1
	report.migrationIds.pop()
	report.resumedAt[resumeStatus] = (report.resumedAt[resumeStatus] ?? 1) - 1
	continue
	}

(Nicer still: a conflicts counter on the report type so skipped races are visible.)

[islandbitcoin]

Treasury resolution is uncached — per-migration mongo reads replacing a memoized value (low)

The old resolver was getFunderWalletId, which memoizes after one ledger lookup. This closure runs findById + (in the BTC-default case ENG-482 explicitly supports) listByAccountId on every call — and it's called per migration at the fee step and per rollback top-up: ~1,200 identical queries across a 600-account run, resolving an id that is invariant for the whole run. A transient mongo blip mid-run now fails a migration step that the memoized value couldn't.

Suggested change

	getTreasuryWalletId:
	deps.getTreasuryWalletId ??
	(async (): Promise<WalletId | ApplicationError> => {
	const funderDefaultWalletId = await funderWalletId()
	const funderWallet = await walletsRepo.findById(funderDefaultWalletId)
	if (funderWallet instanceof Error) return funderWallet
	if (funderWallet.currency === WalletCurrency.Usdt) return funderWallet.id
	const funderWallets = await walletsRepo.listByAccountId(funderWallet.accountId)
	if (funderWallets instanceof Error) return funderWallets
	const usdtWallet = funderWallets.find(
	(wallet) => wallet.currency === WalletCurrency.Usdt,
	)
	if (usdtWallet === undefined) {
	return new CashWalletMigrationFailedError(
	"Funder account has no USDT wallet — create and fund the cutover treasury before running the cutover (ENG-482)",
	)
	}
	return usdtWallet.id
	}),
	getTreasuryWalletId:
	deps.getTreasuryWalletId ??
	(() => {
	let cached: WalletId | undefined
	return async (): Promise<WalletId | ApplicationError> => {
	if (cached !== undefined) return cached
	const funderDefaultWalletId = await funderWalletId()
	const funderWallet = await walletsRepo.findById(funderDefaultWalletId)
	if (funderWallet instanceof Error) return funderWallet
	if (funderWallet.currency === WalletCurrency.Usdt) {
	cached = funderWallet.id
	return cached
	}
	const funderWallets = await walletsRepo.listByAccountId(funderWallet.accountId)
	if (funderWallets instanceof Error) return funderWallets
	const usdtWallet = funderWallets.find(
	(wallet) => wallet.currency === WalletCurrency.Usdt,
	)
	if (usdtWallet === undefined) {
	return new CashWalletMigrationFailedError(
	"Funder account has no USDT wallet — create and fund the cutover treasury before running the cutover (ENG-482)",
	)
	}
	cached = usdtWallet.id
	return cached
	}
	})(),

Also note: .find() takes the first USDT wallet in unsorted listByAccountId order. Today addWalletIfNonexistent dedupes on (type, currency) so duplicates shouldn't exist, but there's no unique index enforcing it — a defensive error when more than one USDT wallet matches would prevent paying from an unfunded duplicate.

[islandbitcoin]

All review findings addressed in b0ac0417. Point-by-point:

• Fee absorption threshold — decoupled as suggested, with one improvement on the TODO: instead of guessing, I measured IBEX's minimum on TEST (no-amount USDT invoices: 600 micros → 400 Bad Request, 650+ → paid). The floor is the LN 1-satoshi minimum, which floats with BTC price — so the constant is a margin, not the measured value: CUTOVER_MIN_PAYABLE_USDT_MICROS = 2_500 (covers 1 sat to ~$250k/BTC, still ¼ cent). The 2500–9999 band reimburses again via the no-amount path, which is live code once more; the 4735-micro spec case is asserted payable.
• Audit false shortfalls — computeCutoverBalanceAudit now grants an allowance of exactly the recorded-but-unpaid fee (feeAmountUsdtMicros with no feeReimbursementPaymentTransactionId); real shortfalls beyond it still flag. Exported + 3 new tests (balance-audit.spec.ts).
• Sub-minimum reverse payment — guard added using the same shared constant; drained/1-cent wallets now fail closed with a clear message instead of a doomed pay + misleading 400. Test added.
• Step-3 boundary band — documented as an accepted tradeoff (comment at the top-up gate): bounded at ~1 cent + fee + dust per account vs. the old behavior of stranding every such account in review.
• State-machine comment — corrected per your text. I kept legacy_zero_verified routing to failed deliberately rather than adding it to AMBIGUOUS_SIDE_EFFECT_STATUSES: those failures are usually transient verify reads and stay auto-retryable, and retry-failed's guards (payment-txn skip, pointer-aware resume — which you verified) make the edges safe. The comment now says exactly that instead of promising what the machine doesn't enforce.
• retry-failed race — skip-and-continue with a new conflicts counter on the report. Test added.
• Treasury resolution — memoized for the run lifetime + fails closed on duplicate funder USDT wallets. Tests for both.

Follow-up notes filed as ENG-485 (IbexError httpCode propagation → === 429 detection everywhere; exporter treasury-USDT gauge) — both touch shared surfaces beyond the cutover and deserve their own regression pass.

12 suites / 68 tests green.