[Feat] Consolidation limits the max fee it will pay

[yorickdowne]

Description

When consolidating "a bunch" of validators, one after the other from the same wallet, valctl sends the escalating fee the contract charges without a ceiling or warning.

Reproduction steps

Send "many" consolidation requests in a short amount of time - in our case under 500 escalated the fee the contract charges to where it drained our wallet.

Current Behavior

Started sending ETH to the consolidation contract without limit, eventually wiping us out. See https://hoodi.etherscan.io/txs?a=0xb6639c4d2c0967414f86919ccdd9af3f76f88ab1&p=5 - it starts with 0 ETH, then sends a bit, and eventually sends 200+ ETH per call.

Expected Behavior

eth-valctl warns the user about high fees as the queue grows, maybe even limits to a max and backs off when fee is higher than that

Additional information

No response

[yorickdowne]

Potentially not a bug but user error.

"that looks like the normal behavior of the fake exponential fee with a growing queue.
I assume you sent a few hundred to thousand consolidations at once, so the queue filled up massively.
The growing fee is a economic limiter that nicely shows it works here :D"

[yorickdowne]

However - I should see and approve the fee up front. Better even have the option to set a max, above which eth-valctl backs off and waits.

So not a bug, but yes an improvement suggestion

[yorickdowne]

Optimal for consolidations is to not use the "N validators as fast as possible" feature of eth-valctl, but instead call it with one source validator at a time. That way it sends one request, waits for it to be included, and then the next request is in the next block. 1 request per block -> no fee increase. That is if I am the only one sending requests. An improvement would be to be able to say "max fee 1 wei, wait for next block and check again otherwise", and that way multiple people sending requests will wait their turn.

That'd be a feature, to be able to specify the maximum fee to pay, and a documentation change: To document not just that multiple validators can be specified via a file, but also that this could escalate the fee, and that a max fee of 1 wei protects against that escalation.

A conservative default for this max fee feature would be 1 wei. At that point eth-valctl could be called safely with a file containing multiple validators, as it would send two, back off for one block, send two again, &c . The exact escalation may be a little different with up to 20 requests sent before fee increases to 2 wei, and a longer back off. This would happen organically, by the simple method of only sending a request if it costs no more than the max fee.

[TobiWo]

Hi @yorickdowne , I was aware of this when implementing the base features but it was a detail I kind of disregarded for the time being. Anyway, it it a nice addition for sure. It is an addition to these issues:

• Add repeating checks for high fees required due to long contract queue #33
• Add CLI command to fetch estimated fee required for EL request #9

Optimal for consolidations is to not use the "N validators as fast as possible" feature of eth-valctl, but instead call it with one source validator at a time.

This can be already achieved while using --max-requests-per-block. I guess you found this setting already.

Summary

1. I would add a separate cli command for checking fees upfront
   • this could be combined with something like --number-of-planned-requests and --max-requests-per-block so the user sees how big the fee would grow
2. I would add something like --max-allowed-contract-fee (default 1 wei)
   • if you set the batch size to e.g. 20 requests per block, I would calculate the fee for the next batch upfront and it would fail (or valctl requests for continuation or cancellation -> would like to keep the control in the hand of the user) if it crosses --max-allowed-contract-fee
3. Improve fee handling documentation

Most of this does not take into account when someone else sends requests to the contract in parallel. However, the max fee would hit here nicely. Furthermore, as you stated, the most conservative and safe way (in terms of fees) to use eth-valctl is with --max-requests-per-block 1 and the new --max-allowed-contract-fee.

Any additions? Did I misunderstood you?

Thanks for raising this.

Best,
Tobi

[yorickdowne]

Yep, that's about the gist of it. I gave you a PR with --max-fee and --max-fee-per-gas. It loops indefinitely until --max-fee constraint can be met, as consolidations should eventually quiet down even when someone else is rapidly consolidating. There's a finite number of consolidations any one entity will run. For --max-fee-per-gas it tries for an epoch before exiting with an error: Gas could stay higher than the given max for a very long time.

I've chosen 1wei and 15gwei as the defaults for either.

This narrowly makes the tool usable safely for our Hoodi consolidations, while stakely also consolidates. We paid around 12,500 Hoodi ETH to learn this lesson 😬 . To give an example, we were consolidating at 1 wei, 1 per block, then stakely jumped in. This escalated to around 2.3 ETH fee for one of their batches, and our very next attempt was over 7,000 ETH. At some point around 1,000 "excess requests", the escalation gets rather brutal. Consolidating without a way to specify max fee is not safe.

The optimal way to consolidate, not taking into account gas savings via EIP-7002, is to run one consolidation per block with a --max-fee 1wei. Sending faster just escalates the fee, rather rapidly in fact. It's no faster to send 63 consolidations and then back off for 30 minutes (a little less is needed, but almost 30), than it is to consolidate 1 per block. But the first method does spend more in fees.

For EIP-7002 it'd probably be similar. Submit when fee is 1 wei; then let the tool wait until it's back down to that again, repeat. That way, it saves on gas, and spends the same on fees. But that's for later, when you have EIP-7002 support.

[TobiWo]

I agree with everything. You know, when you work on these things it is like "Yeah, everybody knows about the exponential nature of these system contract fees. So I can implement the saving conditions later.". But I guess I was wrong. It is a very specific thing and you only know when you go deep into the EIP definition. So it is indeed a must have for eth-valctl. I'm happy that your only lost these 12k ETH on Hoodi 😬 . Sorry for that.

I will have a look into the PR. Thanks again for this.

[yorickdowne]

I have the EIP wrong ofc. EIP-7702, I somehow lost that "7". Stakely uses that to bring sanity to their Ledger withdrawal address: 400 tx at a time, at a low fee. Fee then bumps up in the next block. I stuck their quote, with permission, here: #49 (comment)

[yorickdowne]

"Yeah, everybody knows about the exponential nature of these system contract fees. So I can implement the saving conditions later."

If only.

I had no clue. Same for others who are deep into the Ethereum roadmap. And even those who know it's exponential were a little surprised at how quickly it can escalate: For example with stakely doing 2 batches of 400, this is what my changed eth-valctl logged (note 16 gwei to near 69_000 ETH in 6 blocks):

Block 3032988 | Current contract fee 16.54 gwei exceeds max fee 1 wei — waiting for next block...
Block 3032989 | Current contract fee 15.60 gwei exceeds max fee 1 wei — waiting for next block...
Block 3032990 | Current contract fee 14.71 gwei exceeds max fee 1 wei — waiting for next block...
Block 3032991 | Current contract fee 1783915 gwei exceeds max fee 1 wei — waiting for next block...
Block 3032992 | Current contract fee 1682013 gwei exceeds max fee 1 wei — waiting for next block...
Block 3032993 | Current contract fee 1585932 gwei exceeds max fee 1 wei — waiting for next block...
Block 3032994 | Current contract fee 68986.266236929389527006 ETH exceeds max fee 1 wei — waiting for next block...
Block 3032995 | Current contract fee 65046.736528839120697711 ETH exceeds max fee 1 wei — waiting for next block...

Without a max limit, if you happen to start your own consolidations when someone else is about to post a couple large Ledger batches, it's lights out. Obviously someone shouldn't have 70k in a wallet that's used to consolidate 😅 . But whatever is in there will be taken entirely, as the fee decays down again and hits the value of whatever is in that wallet. Assuming an automated process for a significant number of keys.

The pandaops guys were like "sheeesh". It's one thing to know the exponential fee is there; it's another to see it in action.