[WIP] Alpenglow branch engine: execute-on-receipt replay, wear-first durable store, local cert assembly - #246
Open
7layermagik wants to merge 11 commits into
Open
Conversation
…s plumbing This branch targets Alpenglow clusters exclusively. Delete pkg/forkchoice (vote parsing, confirmed-leaf resolution, skip-path heuristics) and every caller: buffered/confirmation-gated execution, consensus-managed Lightbringer handoff, the classic leaf-bankhash divergence check, and the consensus mode enum (the engine collapses to a single Alpenglow observer constructor). The EpochAuthorizedVoters state plumbing goes end-to-end (no non-forkchoice consumer); old state files with the removed JSON field still load. The fork-tree/branch-overlay replay engine files are deleted with it — the replacement (WorkingSet suffix + certificate-driven switch) lands in the following commits. Net ~-6,000 LOC. TowerBFT clusters remain served by the dev branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ction; WorkingSet suffix State model: ONE canonical durable timeline plus a short in-RAM mutable suffix. The WorkingSet (pkg/accounts) holds executed-but-unrooted slots as a flat O(1)-read map with a per-slot undo journal; PromotePrefix folds the rooted prefix out, EvictFrom unwinds a wrong suffix exactly (randomized model-tested against a naive shadow). Siblings are never materialized as state. Durable commits (pkg/accountsdb) are K-slot batch folds: union-deduped newest-wins deltas stream to ONE sequential segment (appendvec encoding, same read path as bootstrap files) with ONE fsync and ONE atomic Pebble index flip. The manifest is simultaneously the commit record, the index redo log (enables running the index without a WAL), the undo-pointer log, and the carrier of batch bankhashes + resume context — so RecoverFoldState rebuilds the durable frontier from the store itself and a kill -9 no longer forces a snapshot re-bootstrap. Crash matrix (injected panics at every commit stage) tested; fileIds never reused; orphans GC'd against a bootstrap high-water sidecar. The undo-pointer log yields two capabilities: RewindToBatchBoundary restores durable state to any retained fold boundary (late-detected divergence recovery without a snapshot restart; crash-safe via parked manifests, interruptible at any point), and CompactOnce reclaims dead bytes from out-of-horizon segments and bootstrap appendvecs under a pin rule that keeps every in-horizon undo target alive (rewind and compaction compose). Reads retry once on a compaction race instead of panicking. ResumeContext carries the running transaction count (presence-marked) so resume and fork-switch unwinds restore it exactly. State-file writes are power-loss durable (tmp + fsync + rename + dir fsync, the same discipline as the fold manifests) — the epoch stakes persisted at boundaries have no other durable home. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e chain queries Local certificate assembly from raw votor votes (the latency accelerator — footer certs remain the guaranteed input for unstaked observers), built to Agave parity and cross-checked against anza-xyz/alpenglow votor sources and SIMD-0326: thresholds (60/60/60/80/82), vote-to-cert unions (notarize∪notar-fallback, skip∪skip-fallback), per-validator vote budgets (notarize 1 block/slot, notar-fallback 3 distinct blocks), base/fallback bitmap disjointness, fallback-only assembly (base3 with an empty base group), and base2-when-base-only encoding. Pool-assembled certs pass the same production verifier as wire certs; wire codec is validated by agave-votor-messages fixtures. TRUST BOUNDARY: raw votes are unauthenticated network input. Ingest only shape-checks, bounds, and parks votes; dedupe, the vote-budget/equivocation ledger, and disjointness mutate strictly AFTER BLS verification — a bogus vote can neither suppress a victim rank's real vote nor forge equivocation evidence. The vote window anchors to replay-observed progress (never to raw vote slots) with hard per-slot and global memory caps. Verification is lazy and batched (same-payload tallies verify with ONE aggregate pairing, bisecting on failure) but ALWAYS trigger-fresh: the pool folds the involved tallies the moment any Votor fallback-trigger predicate — SafeToNotar(b): notar(b)>=40% OR (notar(b)>=20% AND notar(b)+skip>=60%); SafeToSkip: skip+(notarTotal-topNotar)>=40%, transcribed from Agave votor/src/common.rs — passes on candidate stake but not yet on verified stake, including the cross-tally mixed condition. One fold policy for observer and voting nodes alike; a voting engine reads VerifiedVotorStakes and observes predicate crossings exactly when an eager-verification client would, while sub-trigger tallies cost zero pairings. ChainTracker additions: CertifiedBlockAt (unique-strength or finalized — allocation-free; the switch sweep's oracle), SkipCertifiedAt, WantedBlocks (one most-decisive unobserved repair target per slot), and a monotonic decision version that advances on replay-derived decisiveness (parent links, finalized ancestry, indirect skips), not just cert acceptance. The engine collapses to the Alpenglow observer and wires the pool (emit -> tracker, validator sets, epoch lookup, prune floor). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…riven repair Execute-on-receipt keeps the block source's emission policy (first-assembled candidate emits; certified-skip marking; pre-emission mismatch discard; conflict halt) and adds the consensus-steered correction paths: - RewindForAlpenglowSwitch rewinds the emission frontier when replay unwinds a certificate-contradicted suffix: purges buffered state at/above the switch slot and re-arms turbine toward the certified block id. - Certified skips apply in ANY mode (RPC catchup / pre-handoff included) — a consensus fact survives block-source recreation on a post-switch re-replay, so a certificate-skipped slot is never re-fetched or re-run. - Cert-driven repair: a near-tip loop services ChainTracker.WantedBlocks (certified-but-unobserved, the DECISIVE candidate per slot), pinning the assembler to certified block ids, pulling repair, discarding buffered candidates that carry a contradicted id, and cancelling shred state for certificate-skipped slots — rate-limited one nudge per slot per second. This also keeps re-hinting a certified sibling after a fork switch until its data arrives. The consensus-managed Lightbringer path is gone (deleted with TowerBFT). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…atermark folds Blocks execute the moment they are assembled; certificates never gate execution — they gate PROMOTION TO DISK and drive fork/skip corrections. Dual watermark (the safety keystone): Alpenglow certificates attest block DATA, not execution results, so a trailing verifier is the only execution- correctness oracle. It re-derives compact per-transaction digests (fee, status, balances; CU deliberately excluded) from finalized RPC blocks ~32 slots behind tip and compares against replay's run-local capture registry. Folding is clamped to min(certificate finality, verified watermark) through ONE shared gated path used by both in-loop promotion and the graceful- shutdown flush — a Ctrl+C can never fold a slot the loop would refuse. Promotion re-evaluates every iteration (verified progress alone advances it) and a verifier divergence halts regardless of finality progress, recording persisted evidence that blocks re-folding the disputed slot after restart. Required-verifier + RPC outage stalls folds into the bounded-tail halt: fail-closed by design. Fork switch: a sweeper walks the executed-unfolded window whenever the tracker's decision version advances and reports the first slot whose executed identity a decisive certificate contradicts (wrong sibling or certified skip). The common case unwinds IN RAM — WorkingSet suffix eviction, resume state rebuilt from the retained parent context (the true executed parent, across skipped slots), block source rewound, capitalization/block-height/ transaction-count restored — and the certified version re-executes without a process restart. Guards force the rooted-checkpoint fallback instead when the switch is unsafe in-RAM, each tested: cross-epoch spans, the partitioned- rewards window, missing parent context, and ANY vote/stake-cache write above the rooted watermark (the caches cannot be rolled back, and the resume path's cache reload reads durable, which cannot see retained-suffix writes). A ResumeContext field tripwire forces every new context field to be wired through resume + unwind restoration. Crash/stop-resume completeness: epoch stakes persist to the state file AT the epoch boundary (previously only on graceful shutdown — a hard crash after a boundary forced a snapshot re-bootstrap for the rest of the run); durable promotion HOLDS below the boundary block while partitioned rewards distribute (the RAM-only bookkeeping is not reconstructible mid-window, so a resume must re-run the boundary); the graceful-cancel state save re-runs after the shutdown flush so the state file never trails the store. Promotion folds rooted slots K at a time through the batch committer with the chunk-top resume context riding in each fold manifest (a context-less fold is refused — it would be unrecoverable). rpcclient gains the single-attempt finalized block fetch the verifier uses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Startup: the cluster gate (this build boots network.cluster="alpenglow" only; rooted-durable is forced on), fold recovery BEFORE the bankhash integrity check (manifest replay restores NoSync rows so a hard kill cannot falsely condemn a healthy store), state-watermark reconcile against the store's durable frontier, interrupted-rewind completion (parked manifests resume uniformly regardless of crash point), --rewind-to-slot, the trailing- verifier [verifier] config, fold batch sizing, and opt-in background compaction with validator-safe budgets. Recovery loop: certificate-switch and finality-mismatch re-replays from the rooted checkpoint with a bounded budget; a divergence at/below the durable watermark rewinds the store to the newest retained boundary below it instead of re-bootstrapping (the reason durable state deliberately lags). The retry passes the failed run's serialized epoch stakes through verbatim (raw PersistedEpochStakes JSON, the state-file convention — the consumer json.Unmarshals them directly). Config generators (config init / setup wizard / editor / doctor / migrate) emit Alpenglow-only configs: turbine as the live source (RPC is catch-up/ debug — RPC blocks carry no block ids or footer certs), live [consensus] keys, and TowerBFT clusters marked as dev-branch builds. docs/alpenglow_branch_engine.md replaces the deleted TowerBFT fork-choice doc: the engine model, storage design, Agave-parity certificate semantics, and the voting-readiness contract (single fork choice; the voting engine plugs in above via VerifiedVotorStakes, the cert emit callback, and tracker queries). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rework the 100-slot summary and per-slot lines around the signals an Alpenglow operator actually watches, using Agave terminology: a slot is 'full' when all its data shreds are present and the block is reconstructable (SlotMeta/is_full) — not finalized. Summary (turbine): source; slots/sec with 'behind latest shred: replay N, full N' gaps; skipped + empty blocks (skips annotated with how many had partial shreds); consensus (finalized slot, certificate switches); safety (exec-checked slot from the trailing verifier, promotion holds — counted only when promotion is fully stalled while finality has run a whole fold chunk ahead, so healthy steady state reads 0); shred first/full timing (median/max); repair totals; txns median/p90/max with cu/tx; cu median/p90/max; execution median/p95/max with a >200ms count; efficiency in ms/Mcu; resources (Linux RSS from /proc/self/statm, heap, per-window gc count). RPC-only mode omits every shred metric — nothing is fabricated for non-shred sources. Shred timings are self-anchored: deltas against the fastest cadence-adjusted arrival (400ms nominal), rebased each window to absorb drift, since there is no observable wall-clock slot schedule. Per-slot lines gain shreds(first/full/repair) for shred-sourced blocks and drop wait/vote-split/replay-total from the terminal. Skipped slots align visually — and when the leader DID send shreds before the slot was skipped, the line reports them (shreds(first +18ms, full --, repair 3, partial 12)): 'leader started transmitting then stopped' is a different operator story from 'leader never showed up'. Partial arrivals are snapshotted whenever a never-full slot's assembler state is dropped (reset, prune, cap eviction, non-canonical discard) and retained for the completed-slot window. The old detail (wait, vote/non-vote, COW, memory breakdown, getBlock fetch stats) moves to file-only logs; the tip-stale WARNING stays on the terminal. Plumbing: the turbine assembler stamps each slot's first-shred arrival and full time and counts repair-delivered shreds (the receiver attributes a shred to repair when it matches an outstanding request by responder+nonce), attaching them to the assembled block; monotonic ShredEdges (latest shred slot, highest full slot) and per-slot ShredObservation exposed through the block source; the receiver's LastDataSlot is now a monotonic max so out-of-order packets cannot move the reported edge backward. Tests: percentile/format helpers, self-anchored delta math (fabrication-free for missing full timestamps), and the exact per-slot/skipped line shapes — including the partial-shreds skip case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-mode config profiles Two node types, selected by consensus.mode (one fork choice for both): - "verifying" (default) — non-voting: observe, execute, verify. No keypairs. - "validator" — selectable now, ahead of the voting engine: startup enforces the full voting-deployment shape (identity + vote-account keypairs, turbine block source, gossip entrypoint, Votor QUIC listener) and refuses to start listing anything missing, so validator deployments are provisioned once and stay valid when voting activates. Until then the node runs the verifying pipeline and warns loudly that NO votes are cast. The authorized-withdrawer keypair is deliberately NOT a runtime requirement (keep it offline). Defaults brought current with the Alpenglow-only build: - network.cluster defaults to "alpenglow" (mainnet-beta/testnet/devnet still recognized and pointed at the dev branch until those clusters upgrade). - block.source defaults to "turbine" with the bind address defaulting to 0.0.0.0:8001 — a bare 'mithril run' boots the live shred path. The lightbringer auto-switch keys on explicit-vs-defaulted source rather than the old literal "rpc" default. - Stale flag help/comments fixed (--cluster, --block-source, min_node_version). Config surfaces per node type: - 'config init' (verifying) and 'config init --validator' generate tailored starter configs; the setup wizard asks the node type first, always collects the gossip entrypoint (turbine needs it as much as the sidecar did), asks validator keypaths, and skips the Lightbringer question for validators. - doctor and the dashboard validate consensus.mode and list exactly which validator requirements are missing; the config editor's sidecar-off fallback reverts to turbine, not rpc. - config.example.toml gains an at-a-glance turbine deployment checklist mapping every --flag to its config key; README rewritten for this build: Alpenglow-only intro, both node types with their init profiles, turbine-first flow, honest current-limitations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, switch instrumentation, decisive-oracle completeness Review-response hardening of the fork-choice path, four pieces: ASYNC PROMOTION. CommitBatch (segment write + fsync + index flip, hundreds of ms per K-slot chunk) was the replay loop's one remaining synchronous stall. Promotion is split build/run/apply around a single-flight worker: the loop builds an immutable fold job, a worker runs the durable half, the loop applies completions — all WorkingSet/map mutation stays on the loop thread. Fork switches DRAIN the promoter before evicting (a drained fold that moves the frontier past the switch slot defers to node-level recovery — new 'durable-overlap' fallback reason); the shutdown flush drains then folds synchronously through the SAME gate; every exit applies a drained completion; a discarded-but-committed fold is the already-supported crash-after-commit case. Proof test: force-flush folds through min(finality, verified) and not one slot further. BRANCH-SCOPED STAKE INDEX. stake_pubkeys.idx appends were per-block — durable writes for blocks a certificate switch could still unwind, leaking wrong-fork pubkeys into the index that enumerates the epoch-stakes scan, plus per-block file syscalls on the hot path. Entries are now slot-keyed: flushed (fsync'd) at fold time BEFORE the batch commit (crash = harmless superset, never a scan-corrupting subset), merged into stake scans from RAM meanwhile, dropped with the state on unwind, cleared on replay restart. An order-asserting committer test proves flush-precedes-commit. SWITCH INSTRUMENTATION. tryInLoopUnwind reports WHY it declined (nil-tail / cross-epoch / rewards-window / vote-stake-dirty / missing-context / context-rebuild / durable-overlap); the 100-slot summary splits switches into in-RAM vs fallback with per-reason counts in the file log — the signal for whether the in-RAM engine suffices or a branch-aware state engine is actually needed. Guard-matrix tests assert the exact reasons. DECISIVE-ORACLE COMPLETENESS. The switch sweep's oracles (CertifiedBlockAt / SkipCertifiedAt) are now pinned by a scenario matrix — decisive notarize, fallback-only ambiguity, explicit skip, indirect skip, ancestry finalization, two-decisive conflict handoff — which exposed a real gap: a block finalized only by ancestry that never received its own certificate was invisible to the oracles (blockSlots indexes certified blocks only). A per-slot finalized index makes it decisive and repairable, and the ancestry walk mints a hash-bound stub for an exact-but-untracked parent (the finalized child's header binding is protocol-final). The adversarial corner — replay executed an equivocation twin, the finalized descendant is the only evidence — now takes the cheap in-RAM switch with cert-driven repair fetching the true block, instead of a blind spot ending in the fold-gate halt and a full rooted re-replay. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the cadence-anchored per-slot shred deltas with timing measured
against replay itself — the clock an operator actually cares about:
ready < 0 the block finished assembling |ready|s BEFORE replay asked for
it (pipeline ahead); > 0 means replay waited that long
asm first shred seen -> fully assembled
Computed from the two timestamps the receiver already stamps plus the
NextBlock wait timer replay already had (renamed waitStart -> neededAt),
so there is zero added tracking. The 100-slot summary reports ready
median/worst and asm median/max; the skipped-slot line drops its
anchor-based "first" delta and keeps the seen/repaired counts. The
cadence-anchor machinery (computeShredDeltas, shredAnchorCandidate,
nominalSlotNanos) is removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
General (non-catchup) config hygiene: the log-retention default is now max_age_days = 0 (never delete by age) across the example config and the config/setup generators — retention stays bounded by max_size_mb x max_backups, which is the safer default for debugging a validator. The config.example.toml RPC example comments become generic placeholders rather than naming a specific provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
7layermagik
force-pushed
the
7layer/alpenglow-forkchoice
branch
from
July 6, 2026 05:43
258eb68 to
448e6ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alpenglow branch engine: execute-on-receipt replay, wear-first durable store, local certificate assembly
This PR turns Mithril into an Alpenglow-only node: blocks execute the moment they are assembled, certificates gate durability rather than execution, and the storage layer is redesigned around batched log-structured folds for low NVMe wear and cheap crash recovery. TowerBFT support is deleted (it remains on
dev).Base: the Alpenglow fork-choice layer (koro)
This branch builds on
koro/alpenglow-fork-choice(which supersedes the earliersmcio/alpenglowexperimentation that contributed the initial Alpenglow scaffolding plus the native turbine/gossip/repair path). Koro's layer provides:agave-votor-messagesfixtures.What this PR adds (eleven commits, one per line)
pkg/forkchoice, buffered/confirmation-gated execution, consensus-managed Lightbringer, the consensus mode enum, and the authorized-voters plumbing (~−6k LOC).anza-xyz/alpenglowvotor sources and SIMD-0326). Hard trust boundary: unverified votes never touch dedupe/equivocation/tally state. Verification is lazy-batched (one pairing per same-payload tally) but always trigger-fresh: tallies fold the moment any Votor fallback-trigger predicate (SafeToNotar/SafeToSkip, transcribed from Agave) could pass — one fold policy for observer and future voting nodes alike.--rewind-to-slot, evidence-triggered store rewind in the recovery loop, verifier/fold/compaction config, Alpenglow-only config generators, and the engine doc (docs/alpenglow_branch_engine.md).is_fullterminology): shred timing, repair counts, fork-switch outcomes, consensus/safety watermarks, promotion-hold signal; RPC-only mode fabricates nothing. Detailed debugging moved to file logs.consensus.mode = "verifying"(default, non-voting) or"validator"(selectable now: enforces identity + vote-account keypairs, turbine source, gossip entrypoint, and the Votor QUIC listener at startup, runs verify-only with a loud warning until the voting engine lands). Cluster defaults toalpenglow, block source toturbine;config init [--validator]and the setup wizard generate per-mode configs.min(finality, verified)gate, with a proof test); stake-index appends are now slot-scoped/branch-safe (fold-time fsync'd flush BEFORE the batch commit, unwind drops wrong-fork entries); fork switches instrumented (in-RAM vs fallback with per-reason counts); the sweep's decision oracles pinned by a scenario matrix, which exposed and fixed a real gap: cert-less ancestry-finalized blocks are now decisive and repairable.ready(assembly completion minus when replay asked for the slot; negative = pipeline ran ahead, positive = replay waited) andasm(first shred → full), and the 100-slot summary reports their distributions. Computed from timestamps the receiver already stamps — zero added tracking.max_age_days = 0(never delete by age; retention bounded by size × backups, the safer default for a validator you're debugging), and the RPC example comments inconfig.example.tomlare generic placeholders. Catchup-specific config knobs (repair rate, rpc_fallback, snapshot staleness) are not here — they belong with the catchup work.Next step (required before this can be tested)
Make catchup/repair work end-to-end without depending on RPC block fetch, so the node can be soak-tested against the live Alpenglow cluster in the shreds-only configuration this branch targets. That entire effort — the repair-first-catchup scaffolding and its hardening (turbine prewarm during snapshot build, an on-disk shred spool with prefetch hydration, repair-request pacing/admission control, peer quality scoring, and the catchup drive that fills the resume gap purely from shreds) — is staged on
7layer/catchup-repair(which is based on this branch) and will fold back in once it is validated. It is deliberately kept out of this PR because repair is not yet proven to catch up without the RPC safety net. Until then, unit tests are green but there is no full-cluster soak.To do before merge
storage.index_waloff and enable compaction.koro/alpenglow-fork-choiceand were left untouched here to keep this diff off koro's code:fetchResult.absentOK, the RPC skip-confirmation probe pipeline (confirmSlotAbsent/confirmSlotAbsentViaRPC/shouldProbeAbsentConfirmation/shouldFinalizeSkippedSlot, wired but never invoked),staleBackupResendEvery,lightbringerDialTimeout,rescueStaleWaitingSlot(+staleWaitingSlotRetry), and the exportedForceRPCFallback(test-only). Remove them once the base lands, or coordinate with koro.Verification
-raceclean on the concurrency-heavy ones; crash-matrix tests with injected panics at every fold/rewind stage; randomized WorkingSet model test; pool-assembled certs pass the production wire verifier (incl. fallback-only/empty-base bitmaps).consensus_pool,vote_pool,certificate_builder,common.rs) and SIMD-0326; wire codec against Agave-generated fixtures.pkg/sbpf/pkg/sealevel/pkg/bankhash/pkg/fees/pkg/cuare byte-identical to the base commit (no VM changes).🤖 Generated with Claude Code