Skip to content

fix(InventoryClient): clamp excess L2 withdrawal to liquid on-chain balance#3483

Open
droplet-rl wants to merge 3 commits into
masterfrom
droplet/T90K0AL22-C03GHT4RV42-1781167257-515209
Open

fix(InventoryClient): clamp excess L2 withdrawal to liquid on-chain balance#3483
droplet-rl wants to merge 3 commits into
masterfrom
droplet/T90K0AL22-C03GHT4RV42-1781167257-515209

Conversation

@droplet-rl

Copy link
Copy Markdown
Contributor

Summary

  • InventoryClient#withdrawExcessBalances sized the L2→L1 bridge call off virtual balance (actual + pending inbound rebalance credits + pending L2→L1 withdrawal credits) and submitted that amount without comparing it to the relayer's settled on-chain balance.
  • When the swap-rebalancer has in-flight pending CCTP/OFT/Hyperliquid orders destined for a chain, pendingRebalances[chainId][symbol] inflates that chain's virtual balance, the currentAllocPct - targetPct sizing overshoots what's actually on-chain, and the bridge call reverts at simulation time with ERC20: transfer amount exceeds balance.
  • Trigger still uses virtual balance (so pending inbound credits stop us from withdrawing every cycle until they settle); the transfer amount is now clamped to liquid balance via tokenClient.getBalance(chainId, l2Token). If liquid is zero we skip the cycle and wait for the credits to actually land on-chain.

Production symptom

The zion-across-fast-relayer-rebalancer bot has been emitting MultiCallerClient#LogSimulationFailures every ~5 minutes in #zbot-across-error (e.g. parent ts 1781167257.515209). The reverting tx is the CCTP depositForBurn on the HyperEVM/Arb token messenger at 0x28b5a0e9C621a5BadaA536219b3a228C8168cf5d. Today's snapshot:

  • relayer EOA 0x07ae8551be970cb1cca11dd7a11f47ae82e70e67 on HyperEVM has 134,143.96 USDC on-chain (balanceOf against 0xb88339CB…)
  • the bot's most recent Executed excess L2 inventory withdrawal log proposes 602,758.70 USDC (current alloc 35.33%, target 8%, $2.2M cumulative)
  • the ~644k phantom comes from pendingRebalances[HYPEREVM]["USDC"] — credits from in-flight CCTP/OFT orders that haven't settled

rebalanceInventoryIfNeeded already does the right thing on the L1→L2 side via the unallocatedBalance check (InventoryClient.ts:1109); this PR brings the L2→L1 path in line.

Test plan

  • yarn typecheck
  • yarn lint
  • yarn hardhat test test/InventoryClient.InventoryRebalance.ts — 18 passing, including two new tests:
    • Clamps excess withdrawal to liquid on-chain balance when pending rebalance credits inflate the virtual balance
    • Skips excess withdrawal when there is no liquid on-chain balance
  • Deploy and watch #zbot-across-errorfast-relayer-rebalancer simulation reverts should stop and the bot should fall back to withdrawing whatever's actually on-chain each cycle.

Docs

  • docs/inventory-virtual-balance-model.md gets a new "Triggering vs sizing" section documenting the virtual-trigger-but-liquid-transfer rule so future inventory-side mutations don't regress this.

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ac3b32f1c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/clients/InventoryClient.ts Outdated
Comment on lines +1551 to +1553
const amountToWithdraw = desiredWithdrawalAmount.lte(liquidL2Balance)
? desiredWithdrawalAmount
: liquidL2Balance;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reserve shortfall before clamping liquid withdrawals

When pending rebalance credits make desiredWithdrawalAmount larger than the liquid balance, this clamp can withdraw the entire on-chain balance even if that balance is needed for an outstanding shortfall. getCurrentAllocationPct(..., false) subtracts tokenClient.getShortfallTotalRequirement, but after a large pending credit the computed desired amount can still exceed liquid; e.g. with 100 liquid, 100 shortfall, and a large pending inbound credit, this line chooses 100 and drains the chain, leaving the shortfall unfunded until the pending transfer settles. The liquid cap should reserve the chain's current shortfall (or otherwise avoid withdrawing shortfall-needed liquidity) before submitting the bridge call.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in the follow-up commit.

getCurrentAllocationPct(..., ignoreShortfall=false) discounts shortfall in the virtual world, so the unclamped desiredWithdrawalAmount is shortfall-safe. The bug was that min(desired, liquid) could reach back into liquid that was earmarked for outstanding fills once pending credits inflated desired above liquid. Now sizing off max(0, liquid - shortfall):

  • preserves the shortfall reservation already baked into the virtual-balance math
  • still clamps below virtual so the bridge sim can't revert with ERC20: transfer amount exceeds balance
  • skips the cycle when shortfall fully claims liquid

Added two regression tests covering the partial-shortfall (clamp to liquid - shortfall) and full-shortfall (skip entirely) cases, plus a docs update in inventory-virtual-balance-model.md spelling the rule out.

@droplet-rl

Copy link
Copy Markdown
Contributor Author

@codex addressed the inline feedback on InventoryClient.ts — reserving the chain's outstanding shortfall before clamping the L2 withdrawal so we don't drain liquid that's earmarked for outstanding fills when pending rebalance credits inflate the virtual sizing. Added two regression tests (partial-shortfall clamp + full-shortfall skip) and updated the trigger-vs-sizing section in docs/inventory-virtual-balance-model.md. PTAL.

droplet-rl added a commit that referenced this pull request Jun 11, 2026
…ess L2 withdrawal

`withdrawExcessBalances` clamped the L2->L1 transfer amount to the relayer's
liquid on-chain balance, but the chain's outstanding shortfall (deposits the
relayer has promised to fill but hasn't yet funded) was not reserved during
the clamp. `getCurrentAllocationPct(..., ignoreShortfall=false)` already
discounts shortfall in the *virtual* world, so the unclamped
`desiredWithdrawalAmount` is shortfall-safe. But once large pending inbound
rebalance credits push `desiredWithdrawalAmount` above liquid, the previous
`min(desired, liquid)` could pick liquid that's still earmarked for those
outstanding fills, draining the chain and leaving shortfall unfunded until
the pending credits settle.

Sizing the transfer off `max(0, liquid - shortfall)` instead:
- preserves the shortfall reservation that the virtual-balance math already
  assumed
- still clamps below virtual so the bridge simulation can't revert with
  `ERC20: transfer amount exceeds balance`
- skips the cycle when shortfall fully claims liquid (the next cycle picks
  it up once shortfall fills draw it down or the pending credits arrive)

Adds two regression tests:
- "Reserves outstanding shortfall when clamping excess withdrawal to liquid
  balance" — partial-shortfall case
- "Skips excess withdrawal when shortfall fully claims the liquid on-chain
  balance" — full-shortfall case

Also updates `docs/inventory-virtual-balance-model.md` so the "trigger vs
sizing" section spells the rule out explicitly.

Addresses Codex review feedback on PR #3483.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@droplet-rl

Copy link
Copy Markdown
Contributor Author

Thanks for the re-review.

@droplet-rl droplet-rl force-pushed the droplet/T90K0AL22-C03GHT4RV42-1781167257-515209 branch from 3547773 to a8c53c4 Compare June 11, 2026 10:58
…tfall

`withdrawExcessBalances` sized the L2->L1 bridge call off virtual balance
(actual + pending inbound rebalance credits + pending L2->L1 withdrawal
credits) and handed the result to the bridge adapter without comparing
it to what's actually free to leave the chain. Two failure modes:

1. Pending inbound rebalance credits haven't settled yet — the bridge's
   transferFrom reverts at simulation with `ERC20: transfer amount
   exceeds balance`. This is the symptom that's been spamming
   #zbot-across-error since ~2026-05-23.
2. The chain has an outstanding shortfall (deposits the relayer
   promised to fill but hasn't funded yet). Pulling liquid back to L1
   concurrently with promised fills either starves those fills or
   undoes a rebalancer that's already moving funds in to cover the
   shortfall.

Fix:
- Skip the withdrawal entirely while the chain has any outstanding
  shortfall.
- Otherwise, cap the desired withdrawal at the relayer's liquid
  on-chain balance for this token.

Trigger stays virtual-aware (so once the inbound settles or the
shortfall is consumed by fills we re-evaluate next cycle), but the
transfer is now bounded by what's actually free to send.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@droplet-rl droplet-rl force-pushed the droplet/T90K0AL22-C03GHT4RV42-1781167257-515209 branch from a8c53c4 to 2328f72 Compare June 11, 2026 11:08
Comment thread src/clients/InventoryClient.ts Outdated
Comment on lines +1550 to +1552
// Cap the desired withdrawal at the relayer's liquid on-chain balance. Sizing off
// virtual balance would revert at simulation because pending inbound rebalance
// credits haven't actually settled on-chain yet.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is wrong - doing so would leave the operator with 0 tokens on this chain, whereas the intention is to rebalance down to the desired threshold.

Therefore, the withdrawal amount should reduce the liquid token balance down to the target allocation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in the follow-up commit.

Replaced min(desired, liquid) with min(desired, max(0, liquid - targetAmount)), so the post-withdrawal liquid balance stays at the chain's target allocation instead of going to zero. Still bounds the transfer below settled liquidity (so the bridge call doesn't revert at simulation), but no longer strips the operational reserve when the desired (virtual-derived) amount exceeds liquid.

Tests updated to assert the new clamp:

  • Caps the withdrawal at liquid-minus-target when pending rebalance credits inflate the virtual balance — binding clamp, expected = liquid - target
  • Skips the withdrawal when liquid balance is at or below target allocation — degenerate case where pending credits trigger the threshold but liquid is already inside the target reserve

Kept the skip-on-shortfall guard since that's a separate concern (don't withdraw at all while promised fills are outstanding). Happy to also fold that into the unified clamp (liquid - max(target, shortfall)) if you'd prefer — let me know.

…et, not raw liquid

Per @pxrl's review, clamping the L2->L1 transfer at `liquidL2Balance`
could drain the chain to zero when `desiredWithdrawalAmount` (sized off
virtual balance) exceeds what's actually settled on-chain. The intent of
`withdrawExcessBalances` is to bring the chain *down to* its target
allocation, not to strip it of its operational reserve.

Replace `min(desired, liquid)` with `min(desired, max(0, liquid - target))`:
- still bounds the transfer by what's settled on-chain (so the bridge
  call doesn't revert at simulation with `ERC20: transfer amount exceeds
  balance` when pending inbound rebalance credits inflate the virtual
  sizing)
- additionally keeps the chain's target allocation in liquid form, so
  the operator's structural reserve isn't drained and in-flight fills
  aren't starved between this withdrawal and the pending credits actually
  settling

The existing skip-on-shortfall guard stays in place (separate concern:
don't withdraw at all while promised fills are outstanding).

Tests updated/added:
- `Caps the withdrawal at liquid-minus-target when pending rebalance
  credits inflate the virtual balance` — covers the binding clamp case
  (liquid > target, pending pushes desired above liquid - target)
- `Skips the withdrawal when liquid balance is at or below target
  allocation` — covers the degenerate case where pending credits trigger
  the threshold but liquid is already inside the target reserve

Docs: re-add the "Triggering vs sizing" section to
`inventory-virtual-balance-model.md` so the rule is documented at the
inventory level.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@droplet-rl

Copy link
Copy Markdown
Contributor Author

Addressed @pxrl's inline feedback. The clamp now caps the withdrawal at liquid - targetAmount rather than full liquid, so the chain retains its target allocation in liquid form instead of being drained to zero when virtual sizing inflates desired past liquid. Tests updated to cover the binding clamp case + the "liquid already at/below target" skip case. Asked in-thread whether the existing skip-on-shortfall guard should fold into the same clamp or stay as a separate guard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants