Describe the bug
The sequencer fails to update the L1 block tip often enough when constructing L2 blocks, causing L2 blocks to jump from L1 block N to N+2 approximately 20% of the time. When building a block at a higher L1 tip height, the sequencer will update the blockhash ring buffer at position (N+2) - 1 = N+1 with the L2 block hash of the previous block. This means replacing the blockhash in ring buffer slot N never happens, and will stay as a stale L2 block hash from 256 blocks ago. This can stack and multiply over time to lead to stale L2 block hashes that are 256, 512, 768, etc. L1 blocks stale, potentially hours old.
Vulnerability Details
The sequencer updates the L1 block tip in a loop in header_reader.go#L99, however it only does so every 15 seconds, which is configured to be the default.
|
PollInterval: 15 * time.Second, |
|
timedCtx, cancelFunc := context.WithTimeout(ctx, s.config().PollTimeout) |
This means that 20% of the time, 15 seconds will elapse and an entire L1 block header will be missed, when it does update, the L1 block tip in sequential L2 block numbers will jump from N to N+2. As the sequencer begins constructing the next block, it detects that the L1 block tip has been increased and therefore has to update the blockhash ring buffer with the last L2 block hash of that L1 tick. However, it calculates which slot to put it in by subtracting 1 from the new L1 block tip, such that (N+2)-1 = N + 1, and the ring buffer slot for block N is never touched, instead retaining whatever value was in that slot previously.
|
state.Restrict(state.Blockhashes().RecordNewL1Block(l1BlockNumber-1, prevHash, state.ArbOSVersion())) |
This also obviously taints the slot N+1 that was filled in, since it got a block hash which corresponds to the last tick, ideally it should get a synthetic blockhash and N gets the correct hash inserted.
RecordNewL1Block is meant to catch skipped L1 ticks and fill them in with a synthetic block hash, however this only works for gaps 2 and above, which is broken by the bad code in internal_tx.go
|
for nextNumber+1 < number { |
Impact Details
Because of the 15 second - 12 second disparity with L1 tip updates, the frequency of this bug is not only often, its extremely predictable, since 92% of the time the gap between two missed ring buffer updates will have a gap of 4-6 blocks from each other. And given this blockhash is already known, it is easy to predict with high accuracy ahead of time what a given blockhash will be and when, even when in the eyes of a smart contract using blockhash the block hash should be unknowable to everyone but the sequencer.
I wouldn't consider this a major threat to funds -- it is stated in the docs that blockhash should not be used as a source of randomness. However many protocols do use it in the absence of a reputable VRF provider. Often this is in conjunction with a keeper-operated hash chain to mix in with a future block hash. For example, on RobinHood chain the protocol StockRip, which has over 40 ETH in vulnerable deposits currently. https://stockrip.com/docs https://robinhoodchain.blockscout.com/address/0x32E8D5b0b8643dC002864a2F5e4481E59eb714CB
Knowing the blockhash mixin (i.e. being the operator of keeper of a protocol) ahead of time could allow a malicious operator in many contracts to steal funds that rely on randomness, such as in lottery type systems, as is the case for StockRip.
Outside of randomness, having inaccurate blockhashes on L2 could be an issue for any off chain program using them, as well as smart contracts which may use them as a means to show inclusion or exclusion of data on that chain given a merkle proof or zero knowledge proof. The block hash being so unexpectedly stale when using for example blockhash(block.number-1) could lead to complex attacks where a contract believes a proof is currently true, however the proof uses block hashes hours older than they expected. This is different to synthetic or missed block hashes, which can be easily detected and ignored by contracts, having incorrect hashes is a more severe issue.
This issue is present on every Arbitrum Nitro chain I've looked at, including Arbitrum One, Robinhood chain, Xai, etc.
Remediation
Immediately, a flag can be added to any nitro sequencer nodes, --parent-chain.connection.poll-interval, which overrides the 15 second HTTP RPC default to make it poll more often. Alternatively, the parent chain can be connected via a websocket which the sequencer should subscribe to automatically, making L1 updates instantaneous.
In addition, the faulty code in internal_tx.go should be updated to make sure to miss these types of gaps and fill them in with synthetic block hashes properly.
|
state.Restrict(state.Blockhashes().RecordNewL1Block(l1BlockNumber-1, prevHash, state.ArbOSVersion())) |
Proof of Concept
https://gist.github.com/SanLeo461/fa62e6333758144893ff633916ee73ba
To run, use bun install && bun run poc.ts. The script simply fetches the last 256 block hashes using blockhash(n) from the chain using an eth_call with a contract override. Then it fetches corresponding block headers for the block hashes and checks their age, showing that ~20% of them are >=256 behind, and a further ~20% are off by 1, due to the double slot tainting described above.
Describe the bug
The sequencer fails to update the L1 block tip often enough when constructing L2 blocks, causing L2 blocks to jump from L1 block N to N+2 approximately 20% of the time. When building a block at a higher L1 tip height, the sequencer will update the blockhash ring buffer at position (N+2) - 1 = N+1 with the L2 block hash of the previous block. This means replacing the blockhash in ring buffer slot N never happens, and will stay as a stale L2 block hash from 256 blocks ago. This can stack and multiply over time to lead to stale L2 block hashes that are 256, 512, 768, etc. L1 blocks stale, potentially hours old.
Vulnerability Details
The sequencer updates the L1 block tip in a loop in header_reader.go#L99, however it only does so every 15 seconds, which is configured to be the default.
nitro/util/headerreader/header_reader.go
Line 99 in a618155
nitro/util/headerreader/header_reader.go
Line 310 in a618155
This means that 20% of the time, 15 seconds will elapse and an entire L1 block header will be missed, when it does update, the L1 block tip in sequential L2 block numbers will jump from N to N+2. As the sequencer begins constructing the next block, it detects that the L1 block tip has been increased and therefore has to update the blockhash ring buffer with the last L2 block hash of that L1 tick. However, it calculates which slot to put it in by subtracting 1 from the new L1 block tip, such that (N+2)-1 = N + 1, and the ring buffer slot for block N is never touched, instead retaining whatever value was in that slot previously.
nitro/arbos/internal_tx.go
Line 98 in a618155
This also obviously taints the slot N+1 that was filled in, since it got a block hash which corresponds to the last tick, ideally it should get a synthetic blockhash and N gets the correct hash inserted.
RecordNewL1Block is meant to catch skipped L1 ticks and fill them in with a synthetic block hash, however this only works for gaps 2 and above, which is broken by the bad code in internal_tx.go
nitro/arbos/blockhash/blockhash.go
Line 56 in a618155
Impact Details
Because of the 15 second - 12 second disparity with L1 tip updates, the frequency of this bug is not only often, its extremely predictable, since 92% of the time the gap between two missed ring buffer updates will have a gap of 4-6 blocks from each other. And given this blockhash is already known, it is easy to predict with high accuracy ahead of time what a given blockhash will be and when, even when in the eyes of a smart contract using blockhash the block hash should be unknowable to everyone but the sequencer.
I wouldn't consider this a major threat to funds -- it is stated in the docs that blockhash should not be used as a source of randomness. However many protocols do use it in the absence of a reputable VRF provider. Often this is in conjunction with a keeper-operated hash chain to mix in with a future block hash. For example, on RobinHood chain the protocol StockRip, which has over 40 ETH in vulnerable deposits currently. https://stockrip.com/docs https://robinhoodchain.blockscout.com/address/0x32E8D5b0b8643dC002864a2F5e4481E59eb714CB
Knowing the blockhash mixin (i.e. being the operator of keeper of a protocol) ahead of time could allow a malicious operator in many contracts to steal funds that rely on randomness, such as in lottery type systems, as is the case for StockRip.
Outside of randomness, having inaccurate blockhashes on L2 could be an issue for any off chain program using them, as well as smart contracts which may use them as a means to show inclusion or exclusion of data on that chain given a merkle proof or zero knowledge proof. The block hash being so unexpectedly stale when using for example blockhash(block.number-1) could lead to complex attacks where a contract believes a proof is currently true, however the proof uses block hashes hours older than they expected. This is different to synthetic or missed block hashes, which can be easily detected and ignored by contracts, having incorrect hashes is a more severe issue.
This issue is present on every Arbitrum Nitro chain I've looked at, including Arbitrum One, Robinhood chain, Xai, etc.
Remediation
Immediately, a flag can be added to any nitro sequencer nodes, --parent-chain.connection.poll-interval, which overrides the 15 second HTTP RPC default to make it poll more often. Alternatively, the parent chain can be connected via a websocket which the sequencer should subscribe to automatically, making L1 updates instantaneous.
In addition, the faulty code in internal_tx.go should be updated to make sure to miss these types of gaps and fill them in with synthetic block hashes properly.
nitro/arbos/internal_tx.go
Line 98 in a618155
Proof of Concept
https://gist.github.com/SanLeo461/fa62e6333758144893ff633916ee73ba
To run, use bun install && bun run poc.ts. The script simply fetches the last 256 block hashes using blockhash(n) from the chain using an eth_call with a contract override. Then it fetches corresponding block headers for the block hashes and checks their age, showing that ~20% of them are >=256 behind, and a further ~20% are off by 1, due to the double slot tainting described above.