feat(replay): step a past slot range through dora as if it were live - #842
Open
pk910 wants to merge 3 commits into
Open
feat(replay): step a past slot range through dora as if it were live#842pk910 wants to merge 3 commits into
pk910 wants to merge 3 commits into
Conversation
loadEpochStats read epochStats.dependentState eight times across a loadState call that can run for seconds, while runCachePruning writes nil into that same field from another goroutine. getEpochStatsByEpoch releases cacheMutex before the write, so nothing serialises the two: a prune landing mid-load turns the next read into a nil dereference, the loader goroutine panics, and the epoch's stats are never completed. Read the field once into a local and work from that, at every check-then-read site in the file. The window is normally too small to hit, since a state comes off a local node in milliseconds. It reproduces reliably against a replayed chain, where the state arrives over the network and the epoch being loaded can fall out of the in-memory window before the load returns, leaving epochs with a row but zeroed validator counts and no participation.
dora-replay serves a fake beacon/execution node pair backed by a real upstream
and drives a virtual clock the explorer follows, so a past slot range unfolds
slot by slot with pause, step, seek and play while the normal UI stays live and
inspectable. No indexer, service or handler code is aware of it, which is the
point: a run has to exercise the real code paths.
The explorer's whole share is the clock. ChainState gains an optional replay
clock that polls the control server and interpolates between polls; CurrentSlot
follows it, and a small ticker replaces ethwallclock, which reads time.Now()
internally and cannot be paused. Genesis stays real, so SlotToTime keeps
returning true historical times and only "now" moves. Page timestamps use the
same clock. The control panel is side-loaded from the replay process, so the
layout carries a script tag and nothing else.
Notes on the fake nodes:
- States are read from tracoor first and blocks are not. Beacon nodes prune
all but the most recent states (a node tagged archive: true 404s a state
two days old), while tracoor keeps every one it sampled. Blocks are the
other way round: the node answers in one round trip where a tracoor read
costs a lookup plus a download, and routing blocks through it tripled the
explorer's per-block indexing time. Tracoor artifacts arrive zstd-compressed
and without Eth-Consensus-Version, both of which have to be handled or the
explorer cannot decode them.
- states/head/finality_checkpoints is answered locally. Reading it upstream
forces the node to load the head state, which for a replayed head is
expensive at best and pruned away at worst.
- Chain events are never dropped: the hub waits for its subscriber. That is
what makes "advance as fast as upstream allows" self-throttle to what the
explorer can actually index. Control status events are the opposite, so a
stalled browser tab can never hold the replay up.
- Halfway through each slot the driver waits for any beacon state still on
its way to the explorer, freezing the clock while it waits. The wait costs
real time and no virtual time, so the explorer does not end up several
slots behind the slot the replay claims to be at.
Docs in replay/README.md, worked config in replay/example-config.yaml.
The state gate froze the clock for the whole of a beacon state read, so the driver sat in awaitStateLoads and emitted no block or head events at all while it lasted — several seconds per epoch boundary, and far longer when reads stacked up. That was the wrong shape. The explorer loads states on a different goroutine than it indexes blocks on, so a slow read does not stop it from processing blocks; it only stops it from finishing that epoch's stats. Holding the replay for the read therefore idles the block indexer for no benefit. Allow the replay to run stateLoadLeadSlots further while a read is in flight and only hold beyond that, so blocks keep flowing through a multi-second read while the clock still cannot run away. Not the cause, for the record: the proxy does not serialize requests behind a long state read. Measured through it, a cold block fetch during a cold 17MB state fetch takes 0.124s against a 0.37s idle baseline, and headers stay at 0.13s with three concurrent state fetches in flight.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
dora-replay: a fake beacon/execution node pair backed by a real upstream, driving a virtual clock so a past slot range unfolds slot by slot with pause, step, seek and play, while the normal Dora UI stays live and inspectable.It was built to debug the Gloas builder-deposit work against devnet-8, and it has already earned its keep — every deposit fix in the companion PR was found by replaying the fork transition and diffing the projection against the real chain.
What the explorer contributes
The clock, and nothing else.
ChainStategains an optional replay clock that polls the control server and interpolates between polls;CurrentSlot()follows it, and a small ticker replacesethwallclock, which readstime.Now()internally and cannot be paused. Genesis stays real, soSlotToTimekeeps returning true historical times — only "now" moves.No indexer, service or handler code is aware of the replay. That is the point: a run has to exercise the real code paths.
The control panel is side-loaded from the replay process, so the page layout carries a script tag and a single
ReplayControlUrlfield, and the UI itself — markup, styling, API calls — lives in the replay binary and can be changed without rebuilding or restarting the explorer.Notes on the fake nodes
archive: true404s a state two days old — while tracoor keeps every one it sampled. Blocks are the other way round: the node answers in one round trip where a tracoor read costs a lookup plus a download; routing blocks through tracoor tripled per-block indexing time. Tracoor artifacts arrive zstd-compressed and withoutEth-Consensus-Version, both of which have to be handled or the explorer cannot decode them.states/head/finality_checkpointsis answered locally. Reading it upstream forces the node to load the head state, which for a replayed head is expensive at best and pruned away at worst.Also included
fix(indexer): read dependentState once per epoch stats load— a pre-existing data race the replay reproduces reliably.loadEpochStatsreadepochStats.dependentStateeight times across a multi-secondloadStatewhilerunCachePruningwritesnilinto that same field from another goroutine, with nothing serialising the two. It normally never fires (a state comes off a local node in milliseconds); against a replayed chain it left epochs with a row but zeroed validator counts. Kept as its own commit so it can ship separately.Docs
replay/README.md, worked config inreplay/example-config.yaml. Known fidelity limits are documented there — no reorg replay, intra-slot timing approximated, real timers keep running while paused.