Skip to content

feat(cutover): rollback handlers, reverse payment path, and admin surface (ENG-401)#431

Merged
islandbitcoin merged 2 commits into
mainfrom
feat/eng-401-cutover-rollback
Jul 4, 2026
Merged

feat(cutover): rollback handlers, reverse payment path, and admin surface (ENG-401)#431
islandbitcoin merged 2 commits into
mainfrom
feat/eng-401-cutover-rollback

Conversation

@islandbitcoin

Copy link
Copy Markdown
Contributor

Implements ENG-401 — makes the ENG-364 rollback contingency plan executable. Until now rollback_started/rolled_back existed in the state machine but nothing could enter or process them: the cutover runbook's "roll back from snapshot if reconciliation drifts by 1 cent" was unexecutable. This is a go/no-go gate for cutover Phase 1 (ENG-461).

The 6 prerequisites → what shipped

# Prerequisite Implementation
1 Rollback state handlers + legal transitions Every forward/failed/requires_operator_review/complete status → rollback_started; self-loop persists sub-step artifacts; executeCashWalletMigrationRollbackStep (new rollback-worker.ts)
2 Reverse payment path No-amount invoice on the legacy USD wallet paid from the USDT wallet with the exact forward amount — precise mirror of the forward move, incl. stale-invoice refresh and IBEX rate-limit retry (reuses runtime services)
3 previousDefaultWalletId admin read access New cashWalletMigrations admin query returns full migration records incl. pointer + rollback audit fields
4 Batch rollback repository query CashWalletCutoverRepository.listMigrationsByStatuses (status-set driven; powers request + batch execution)
5 CashWalletCutoverState enum ROLLED_BACK added (config-level terminal state; public SDL change is an additive enum value only — non-breaking)
6 Admin rollback mutation cashWalletCutoverRollback: single-account (accountId) or full-run, dryRun, idempotent (already-rolled-back = skips), resumable; admin-only

Plus CLI: rollback-request / rollback-batch / rollback-complete on scripts/cash-wallet-cutover.ts.

Rollback sequence (field-driven, resumable — mirrors forward's field-presence guards)

  1. Pointer restore — only if the account still defaults to USDT; restores previousDefaultWalletId (legacy USD wallet as constructive fallback)
  2. Reverse balance move — pays back exactly destinationAmountUsdtMicros; fails closed (requires_operator_review) if the USDT balance no longer covers it (user transacted post-cutover)
  3. Treasury shortfall top-up — restores the user to sourceBalanceUsdCents within a 1-cent LN-dust tolerance; hard no-double-pay guard (still short after one top-up → operator review, never a second automatic payment)
  4. Finalizerolled_back (+rolledBackAt)

Pre-money migrations (no forward payment ever sent) short-circuit straight to rolled_back.

Acceptance criteria mapping

  • ✅ All 6 prerequisites built; unit-tested (11 new tests; all 7 cutover suites pass, 34 total)
  • ✅ Reverse payments traced by account, migration id, run id (cwco-rb:<runId>:<id>:move|shortfall memos), and operator (rollbackRequestedBy/rollbackReason)
  • ✅ No irreconcilable balances: exact-amount reverse + tolerance-bounded treasury top-up, both verified against live balances before finalizing
  • ✅ Unexpected states fail closed into requires_operator_review (batch runner override: rollback failures are never plain failed)
  • skipped_already_migrated accounts left in USDT (transition rejected by the state machine — unit-tested)
  • ✅ Admin-only (registered in the admin schema; nothing added to public GraphQL — public SDL diff is the enum value only)
  • ⏳ Staging (TEST) end-to-end verification: planned as part of the ENG-461 cutover rehearsal — single-account rollback per migration state + full-run rollback, per the acceptance list

Validation

tsc --noEmit clean (both configs) · eslint clean · yarn test:unit cutover suites 34/34 · SDL regenerated (write-sdl), admin schema gains the new surface

Linear: ENG-401 · builds on ENG-345 (forward orchestration) · implements ENG-364 modes 3A (single account), 3C (full-run pre-complete), 3D (post-complete)

🤖 Generated with Claude Code

…n surface (ENG-401)

Makes the ENG-364 rollback contingency plan executable. The state machine
had rollback_started/rolled_back statuses but nothing could enter or
process them — the runbook's "roll back from snapshot on 1-cent drift"
promise was unexecutable.

State machine: every forward/failed/review/complete status may now enter
rollback_started (skipped_already_migrated and rolled_back excluded by
design); rollback_started self-loops to persist sub-step artifacts and
fails closed into requires_operator_review.

Rollback worker (rollback-worker.ts): field-driven resumable executor
mirroring the forward pipeline's field-presence guards —
  1. restore default-wallet pointer (previousDefaultWalletId, captured at
     forward flip; legacy USD wallet as constructive fallback)
  2. reverse the balance move: no-amount invoice on the legacy USD wallet
     paid from the USDT wallet with the exact forward amount
     (destinationAmountUsdtMicros); fails closed if the USDT balance no
     longer covers it (user transacted post-cutover)
  3. treasury tops up any legacy USD shortfall vs sourceBalanceUsdCents
     (1-cent dust tolerance; hard no-double-pay guard)
  4. finalize to rolled_back with rolledBackAt
Pre-money migrations short-circuit straight to rolled_back. Reverse
payments traced by account, migration id, run id (cwco-rb memos), and
operator (rollbackRequestedBy/reason audit fields).

Orchestration (rollback.ts): requestPrimaryCashWalletRollback (single
account or whole run, dry-run, idempotent — already-rolled-back are
skips), runPrimaryCashWalletRollbackBatch (reuses forward batch
locking/stale-lock recovery; all failures marked
requires_operator_review), completePrimaryCashWalletRollback (flips the
cutover config to the new rolled_back state only when nothing is
mid-flight or in review).

Admin GraphQL: cashWalletCutoverRollback mutation (admin-only),
cashWalletMigrations query exposing full migration records incl.
previousDefaultWalletId and the rollback audit trail, ROLLED_BACK added
to CashWalletCutoverState (additive; public SDL change is enum-value
only). CLI: rollback-request / rollback-batch / rollback-complete.

Tests: 11 new unit tests (rollback-worker.spec.ts) covering request
semantics, every executor path (short-circuit, pointer restore, reverse
pay, fail-closed insufficiency, resume, shortfall top-up, dust tolerance,
no-double-pay), and decimal-cent shortfall math. tsc + eslint clean;
all 7 cutover suites pass (34 tests).

Linear: ENG-401 (implements ENG-364 modes 3A/3C/3D)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@linear

linear Bot commented Jul 4, 2026

Copy link
Copy Markdown

ENG-401

ENG-364

…ing, pointer-restore gate

Review fixes on the ENG-401 rollback implementation:

1. (High) completePrimaryCashWalletRollback could flip the run-level
   config to rolled_back after a single-account rollback while every
   other migration remained `complete`. Completion now requires the
   ENTIRE run to be accounted for: zero migrations outside
   {rolled_back, skipped_already_migrated} — checked via
   listMigrationsByStatuses over every other status, with the offending
   migration named in the error. Single-account rollbacks never complete
   the run.

2. (High) evaluateCashWalletCutoverGuard now handles the rolled_back
   config state explicitly instead of falling through to the in_progress
   branch: rolled_back/absent/not_started -> legacy_usd,
   skipped_already_migrated -> usdt (they were USDT before the run),
   rollback_started -> in-progress error, and anything else (e.g. a
   leftover `complete` under a supposedly rolled-back run) fails closed
   with CashWalletMigrationFailedError rather than guessing a route.

3. (Bonus, same class as 2) the `complete` config state blanket-routed
   every account to usdt — including accounts rolled back AFTER global
   complete (ENG-364 mode 3D), whose funds are back on legacy USD. The
   complete branch now routes rolled_back -> legacy_usd and blocks
   rollback_started, and only then blanket-routes usdt.

4. (Medium) pointer restore is now gated on previousDefaultWalletId
   being present — the forward pointer flip is that field's only writer,
   so its absence proves no flip happened. A pre-money migration whose
   account defaults to USDT (e.g. native USDT-default signup) is no
   longer touched; the fallback flip to legacyUsdWalletId is removed.

9 new unit tests (rollback.spec.ts + rollback-worker.spec.ts): all 8
cutover suites pass, 44 total. tsc + eslint clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@islandbitcoin

Copy link
Copy Markdown
Contributor Author

Review fixes pushed — all three findings addressed, plus a fourth of the same class the review surfaced:

High 1 — completion gating. completePrimaryCashWalletRollback now requires the entire run to be accounted for: zero migrations in any status outside {rolled_back, skipped_already_migrated}, checked via listMigrationsByStatuses with the offending migration named in the error. A single-account rollback can never flip the run-level config; the error message says so explicitly.

High 2 — guard rolled_back handling. Explicit branch instead of fall-through: rolled_back/absent/not_startedlegacy_usd; skipped_already_migratedusdt (USDT before the run, stays USDT); rollback_started → in-progress error; anything else — e.g. a leftover complete under a supposedly rolled-back run — fails closed (CashWalletMigrationFailedError) rather than guessing a route. With High-1 fixed that inconsistency can't legitimately arise, so failing closed is defense-in-depth.

Bonus (same class as High 2) — mode 3D under complete config. The complete state blanket-routed everyone to usdt, including accounts rolled back after global complete whose funds are back on legacy USD. The complete branch now routes rolled_backlegacy_usd and blocks rollback_started before blanket-routing.

Medium — pointer-restore gate. Restore now requires previousDefaultWalletId to be present (the forward flip is that field's only writer, so absence proves no flip happened). The legacyUsdWalletId fallback flip is removed — a pre-money account defaulting to USDT (e.g. native USDT-default signup) is never touched.

On "is partial rollback feasible for a permanent cutover?" Yes, as a T+72h incident tool: (a) routing is per-account (migration-record based) and the phased runbook already operates a mixed fleet for days (Phase 1 internal-only), so one rolled-back account is infrastructurally identical to a not-yet-migrated one; (b) the ENG-364 decision matrix requires it ("$1 discrepancy in any migrated account → rollback that account only") — the alternative would be full-run rollback for one bad account; (c) records are unique per (accountId, runId), so the account re-migrates cleanly in a later run. The limits are now enforced in code: single-account rollback never touches the run config (High-1), and post-complete rollbacks of accounts that have since transacted fail closed to operator review — the mechanical form of the runbook's point-of-no-return. "Permanent" holds at the program level: the exception path's end state is always re-migrate-before-GA, never a standing USD account.

Validation: 9 new tests, all 8 cutover suites green (44 total), tsc + eslint clean.

@islandbitcoin islandbitcoin merged commit a43b321 into main Jul 4, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants