Add payload timeliness committee duty flow - #429
Open
AntiD2ta wants to merge 20 commits into
Open
Conversation
AntiD2ta
force-pushed
the
gloas-ptc-duty
branch
4 times, most recently
from
August 19, 2026 09:04
83be146 to
9b8206e
Compare
AntiD2ta
marked this pull request as ready for review
August 20, 2026 08:33
Schedule payload attestation duties in the Gloas payload window and batch generic signatures per slot.\n\nWire controller, signer, and submitter support for immediate and multinode beacon clients, with payload slot validation and reorg refreshes.\n\nValidation: gosilent test ./..., go build ./..., ./custom-gcl run.
Allow first-success submission cancellation in the multinode test and align controller source with CI lint requirements.
Extract ordered parameter validation and payload attestation scheduling helpers.
Guard the current epoch as well as future ones. The epoch ticker and the following epoch's preparation both reach an epoch, so without this the ticker refetched the duties the preparation had already scheduled and then failed to schedule every one of their jobs. Test the fork on the epoch being scheduled rather than on the current one. The epoch after the fork epoch is prepared from the epoch before it, where the current epoch is still pre-Gloas, so testing the current epoch dropped the first Gloas epoch's duties. On a dependent root change, only reschedule the current slot if its job was cancelled, and skip the refresh while the epoch is unprepared. Rescheduling a job that had already run placed it in the past, where the scheduler runs it immediately and attests to the same slot's payload twice. Sort slots with slices.Sort and derive the scheduler job name through a single helper.
The first node's success cancelled the shared timeout context, aborting the other nodes' in-flight submissions, so on a multinode setup only one beacon node received the messages. Release the context once every submission has finished, matching the execution payload envelope submitter. The fan-out test asserted only that the nodes had been reached at least once between them, which held even when a single node was reached.
The package-level vars asserted nothing and existed only to keep two otherwise unused imports alive. Test the duty's validator index membership with slices.Contains.
Both concrete types define the signing and submission methods unconditionally, so the assertions always succeed and cannot detect a beacon node that does not serve payload attestations. Inactivity before the fork comes from the controller's Gloas fork epoch, not from these checks.
The payload attestation job ran at PAYLOAD_DUE_BPS, the point at which the payload becomes due, and used PAYLOAD_ATTESTATION_DUE_BPS only as a context deadline. payload_present is a question about the payload due time, which the beacon node answers from its own record of when it saw the envelope, so asking at that instant gains nothing and can cost the answer entirely: a beacon node does not serve payload attestation data it does not yet consider final. The other two fields only improve later in the slot, as blob_data_available is evaluated when the data is produced and a block first seen after the payload became due is not seen at all by a vote cast then. Schedule the vote at the attestation deadline instead, with a small grace for a beacon node whose clock trails ours, and run the job's context to the end of the slot so the deadline the job now fires at cannot cut off the signing and submission that follow it. The payload due delay has no remaining use, so remove it with its option.
The retry loop held the attempt count in three places, spelled the retry interval inline, and wrote the terminal submission record from three separate sites -- one of which existed only because the cancellation arm would otherwise return without one. Split the loop from the record. attemptExecutionPayloadEnvelopeSubmission returns the final failure or the context error, and submitExecutionPayloadEnvelope writes the single terminal record on both paths, so a submission that has started cannot exit unrecorded by construction rather than by matching log statements. Backing off at the top of the loop for every attempt after the first drops the guard that suppressed the trailing sleep, leaving one named constant per value. The per-attempt "Failed to submit ... after block publication" warning repeated the error of the attempt record beside it; its attempts_remaining field moves onto that record and the duplicate goes. status="failed" now accompanies every failed terminal record rather than only the cancelled one.
The pre-Gloas timings state what each fraction works out to on a 12-second slot; the Gloas fallbacks lost that when they moved behind dueBPS.
The envelope submitter wrote a terminal record of its own, using the same message the Gloas proposal path already writes for the submission as a whole. With three attempts per envelope a failure logged "Execution payload envelope submission completed" four times, contradicting the single-record contract that path documents. No other Submit* in this package logs at all, so the record goes and its one owner keeps it. The per-provider logger is now built before the semaphore is acquired. A provider that never gets a concurrency slot leaves through the acquire failure, and that record carried neither the provider nor the block root, so the one submission with nothing to show for it was also the one that could not be identified. The client monitor fallback returns to "<unknown>", which serviceInfo hands every other submitter. A configuration name is not an address, and the monitor's provider label is an address space shared with every duty, where a name arrives as a fabricated host. status now carries the outcome alone, with the HTTP status of an API failure beside it in status_code. Reporting the status code in the status field left status="failed" matching only the failures that never reached a node. The provider field takes the beacon_node_address name the rest of the package uses.
DOMAIN_PTC_ATTESTER was resolved without the warning its DOMAIN_BEACON_BUILDER neighbour emits four lines above, so a spec that omits it disabled payload attestation signing silently. The record keeps the error, because domainType distinguishes a domain that is absent, which is expected before the fork, from one of an unexpected type, which is not. Neither half of that behaviour was covered: the warning, and the refusal to sign without the domain. The signing test asserts that no batch reaches an account, so the guard is proven to run before signing rather than beside it.
LogCapture.Write appends under the mutex but Entries returned the backing slice without taking it. The multinode envelope submission test polls Entries from require.Eventually's goroutine while provider goroutines are still logging, so -race aborted the run. Take the mutex in Entries and return a copy of the entries, and cover the concurrent read with a regression test.
Write([]byte(fmt.Sprintf(...))) allocates an intermediate buffer for a writer that fmt can target directly, and the length assertion at the end of the test already proves every write was captured, so the write no longer needs an assertion on its own goroutine.
AntiD2ta
force-pushed
the
gloas-ptc-duty
branch
from
August 21, 2026 09:35
0c268a5 to
a07065c
Compare
Detach payload-attestation-message and execution-payload-envelope fan-out from caller cancellation while retaining the submitter timeout. Add regression tests for the in-flight provider path. Files: multinode payload-attestation and execution-payload-envelope submitters and tests. Verification: race-tested submitter package, full test suite and build with Go 1.25.5, and no new custom lint findings.
AntiD2ta
force-pushed
the
gloas-ptc-duty
branch
from
August 27, 2026 08:22
63b89c4 to
9f6d354
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #428
This branches off
gloas-attestation-deadlinesand targets it, notgloas. #428 supplies thegloasForkEpochfield, the spec-derived deadline machinery, and thedueBPShelper that the payloaddeadlines here are built on, so it should be reviewed and merged first. The diff shown against that
base is this change alone.
The duty flow
A payload timeliness committee duty is a vote on whether the slot's execution payload was revealed on
time.
PTCDutiesis fetched for the epoch, duties are grouped by slot so that every validatorattesting in the same slot shares one job, and each slot's job runs inside the window the fork
defines:
The offset is derived from the chain specification by
obtainPayloadAttestationTiming, in the samebasis points of
SLOT_DURATION_MSthat #428 established. On a 12-second slot it is thespecification's
PAYLOAD_ATTESTATION_DUE_BPS: 7500, so the vote is cast just after 9s and itscontext runs to the end of the slot. The duty is new in Gloas, so it always follows the Gloas slot
duration and has no pre-fork form. Why the vote is cast there, rather than when the payload becomes
due, is recorded in the decision section below.
Signing goes through
SignPayloadAttestationData, which takes the accounts as a slice and returnsone signature per account, so a slot with many attesting validators costs one signing round trip
rather than one per validator.
Submission is versioned and covers all three submitters —
immediate,multinodeandnull— sothe payload vote follows whatever submission strategy is already configured rather than introducing
a separate path.
Decision: when the payload timeliness vote is cast
Recorded here because the choice is not evident from the constants, and an earlier revision of this
branch made the opposite one.
Context
Gloas defines two payload deadlines, and they answer different questions.
PAYLOAD_DUE_BPSPAYLOAD_ATTESTATION_DUE_BPSspecs/gloas/validator.mdis explicit that the second is a window bound and not a target instant: avalidator "should create and broadcast the
payload_attestation_message... within the firstget_payload_attestation_due_ms()milliseconds of the slot". Any moment between seeing the block andthat bound is conformant.
The first is not a scheduling instruction at all — it is baked into the vote's content.
data.payload_presentisTrueonly if aSignedExecutionPayloadEnvelope"was seen beforeget_payload_due_ms()milliseconds into the slot". That is a question about the past, its answer isfrozen once the payload due time passes, and the observation timestamp that answers it belongs to the
beacon node rather than to us.
Two further fields have no time cutoff whatsoever:
data.blob_data_availableisis_data_available(root)evaluated when the data is produced, and the duty is skipped altogether ifno beacon block for the slot has been seen. Both only become more accurate later in the slot.
Why the payload due time was the wrong place
The earlier revision scheduled the job at
PAYLOAD_DUE_BPSand usedPAYLOAD_ATTESTATION_DUE_BPSonly as the job's context deadline, reasoning that the answer is frozen at the payload due time so
there is nothing to gain by waiting. That holds for
payload_presentin isolation and fails for therequest as a whole.
Unavailable— a 503 — when askedbefore the attestation deadline unless the answer is already final, on the stated grounds that
"before the deadline only the final result is safe to return ... otherwise both flags may still
flip". Asking at the payload due time therefore returns nothing in precisely the marginal cases the
vote exists to judge, and our fetch is a one-shot: the duty ends with an error log and no vote.
columns propagating, so a vote cast then reports
blob_data_available: falsefor a payload that wasin fact available.
all, though the specification permits one and the gossip rules accept it.
Both other implementations target the later deadline. Lighthouse's validator sleeps to
duration_to_next_slot + payload_attestation_duebefore fetching; Prysm's waits on thepayload-available event or that same deadline, whichever comes first. We were the outlier, and the
only client fetching at a moment its beacon node may decline to serve.
Decision
Schedule the job at
PAYLOAD_ATTESTATION_DUE_BPSplus a small fixed grace, and run the job's contextto the end of the slot.
the data is final, so firing exactly on the deadline can still be refused by a node whose clock
trails ours. The grace covers that skew and leaves the rest of the slot for the vote to propagate.
Landing just past the nominal deadline is also what both other clients do in practice, since each
fetches at the deadline and only then signs and publishes.
cancel the signing and submission the vote depends on, and the gossip rules accept a payload
attestation for the whole of its slot.
PAYLOAD_DUE_BPSis no longer read. Whether the payload was timely is the beacon node'sdetermination, so
payloadDueDelayandWithPayloadDueDelayare removed rather than left asunused configuration.
Consequences, and what stays open
A fixed deadline is a backstop rather than the best available behaviour: the vote could be cast as
soon as the beacon node reports the payload available, as Prysm's validator does. That needs the
payload-available SSE topic in
go-eth2-client, which does not expose it yet, so aTODOat thescheduling site records it.
This decision depends on
payload_presentbeing answered from a recorded observation timestamprather than from the node's view at request time. That was verified in both implementations —
Lighthouse compares
entry.timestamps.observedagainstget_payload_due(), and Prysm reads aPayloadEarly(root)flag — but it is pinned by neitherbeacon-APIs, which documents neither thefield's meaning nor when it is evaluated, nor by a released version: both were read on unstable
branches that may change before mainnet.
Validation, before signing
Attestrefuses to sign anything it cannot fully account for, rather than signing a best guess:A validator with no account for the duty is skipped rather than failing the whole slot's job, and a
duty with no usable accounts submits nothing at all.
Also in this change
refreshes that already run there: the previous dependent root refreshes this epoch, the current
dependent root refreshes the next. Each refresh cancels the epoch's existing per-slot jobs before
rescheduling them, and leaves the current slot alone when its job has already run.
TestObtainPayloadAttestationTiming(previouslyTestObtainPayloadTimings) asserted that a servedPAYLOAD_DUE_BPS_GLOAStook precedence overPAYLOAD_DUE_BPS. The specification defines no_GLOAS-suffixed form of any payload key, because these deadlines are new in Gloas, so that casetested a key no chain serves. It now asserts that the unsuffixed key holds the deadline, and takes
its expected values from the configuration
beacon.glamsterdam-devnet-8.ethpandaops.ioserves.schedulePayloadAttestationsis split intocanSchedulePayloadAttestations,payloadAttestationsAlreadyScheduledandpayloadAttestationDutieshelpers, to stay within thecomplexity limit. This is a pure extraction with no behaviour change.
Validation
Local Devnet runtime validation — 2026-08-19
vouch:gloasfrom PR heada18564ee03957a1a63093995b4b3a87cc4e7c719and ran it in a fresh-genesis local Devnet-8-derived enclave.503 no canonical shuffling block for slot). At affected slots, all nodes still agreed on and served the canonical block/root, isolating the failure to that endpoint rather than Vouch’s duty scheduling or signing.glamsterdam-devnet-8tags. All were already current; no client image digest changed.