PeerDAS: validator custody applied ~60s after startup; node advertises earliestAvailableSlot it cannot serve, leading to peer-score bans

[barnabasbusa]

Describe the bug

On a fresh-genesis network, validator custody appears to be computed on a ~60 second periodic task instead of at startup. For the first minute after boot the node advertises custody_group_count = 4 (the bare CUSTODY_REQUIREMENT) in its metadata/ENR, even though enough validators are attached to the beacon node from boot to warrant full custody (128 validators × 32 ETH ⇒ all 128 groups).

When the periodic task fires (exactly +60s after startup), custody jumps 4 → 128 and earliestAvailableSlot is set to the current slot — but the node only subscribed to 4 column subnets during the elapsed slots and nothing backfills the missed columns (log shows backfill: Service not enabled). The node therefore advertises column availability (earliestAvailableSlot) that it cannot actually serve: peer data_column_sidecars_by_root requests for those slots return empty responses.

Downstream consequence observed: Lodestar (v1.45.0) peers penalize each empty response (DOWNLOAD_BY_ROOT_ERROR_NO_SIDECAR_RECEIVED, roughly once per slot), cross the peer-score threshold ~7 minutes after genesis, send goodbye (reason: peer score too low) and ban both Prysm peers. On our small 4-node devnet this permanently partitioned the network into a Lodestar island and a Prysm island (50/50 validators, so no finalization on either side).

Timeline of the failing run (6s slots, minimal preset, Fulu at epoch 0):

08:35:01  prysm starts    Custody info earliestAvailableSlot=0 groupCount=4
08:35:01  backfill: Service not enabled / Marked as complete
08:35:32  genesis (slot 0)
08:36:01  (+60s)          Custody info earliestAvailableSlot=4 groupCount=128
08:36-08:42  lodestar peers: NO_SIDECAR_RECEIVED on data_column_sidecars_by_root
             for slots 5..39, ~every slot, from both prysm peers
08:40:49  lodestar refreshes prysm metadata, sees custodyGroupCount=128
          (requests keep returning empty regardless)
08:42:44  lodestar: goodbyeAndDisconnect reason=Peer score too low → ban

Control run showing it is a race between the 60s tick and genesis — same setup, but enclave startup took longer, so the tick fired 1 second before genesis; the network was completely healthy:

09:01:47  prysm starts    Custody info earliestAvailableSlot=0 groupCount=4
09:02:47  (+60s)          Custody info earliestAvailableSlot=0 groupCount=128
09:02:48  genesis (slot 0)
          → no missed columns, no empty responses, no penalties, no partition

Expected behavior — any of:

• compute validator custody at startup (the validators are already attached before genesis), or
• set earliestAvailableSlot to the slot where full custody actually began rather than claiming already-elapsed slots, or
• backfill the missed columns before advertising the new custody count.

Severity note: probably not urgent — on restarts the DB restores previously custodied columns, so this mainly affects fresh syncs with validators attached and new networks/devnets (where genesis lands within ~60s of process start). Filing so it's on record.

Has this worked before in a previous version?

Unknown — first observed on v7.1.8 with PeerDAS (Fulu) active at genesis.

🔬 Minimal Reproduction

1. Kurtosis ethereum-package enclave, minimal preset, Fulu at epoch 0, 6s slots: 2× lodestar/geth + 2× prysm/geth, 512 genesis validators (128 per node ⇒ everyone qualifies for full validator custody), blob spammer (spamoor) enabled.
2. Arrange for genesis to occur shortly (≲1 min) after the Prysm beacon nodes start — with this small participant count the default enclave startup does this naturally.
3. Watch Prysm's DEBUG db: Custody info lines: groupCount=4 at startup, groupCount=128 exactly 60s later, earliestAvailableSlot set to the then-current slot.
4. Watch the Lodestar peers accumulate DOWNLOAD_BY_ROOT_ERROR_NO_SIDECAR_RECEIVED against the Prysm peers and disconnect them with Peer score too low a few minutes later; Prysm logs Received a goodbye message ... reason=peer score too low from the Lodestar peers.

Error

# prysm (cl-3), failing run
[08:35:01.13] DEBUG db: Custody info earliestAvailableSlot=0 groupCount=4
[08:35:01.13]  INFO backfill: Service not enabled
[08:35:01.13]  INFO backfill: Marked as complete
[08:36:01.14] DEBUG db: Custody info earliestAvailableSlot=4 groupCount=128
[08:42:31.53] DEBUG sync: Received a goodbye message isRateLimited=false peer=16Uiu2HAmSFGp... reason=peer score too low
[08:42:44.08] DEBUG sync: Received a goodbye message isRateLimited=false peer=16Uiu2HAmKmnJ... reason=peer score too low
[08:46:01.15]  INFO p2p: Connected peers inboundQUIC=1 inboundTCP=0 outboundQUIC=0 outboundTCP=0 target=70 total=1

# lodestar (cl-1) view of the same prysm peer
[08:35:01] onStatus peerId=16...cLJokZ custodyGroupCount=4 hasAllColumns=false matchingSubnetsNum=0 custodyGroups=[43, 82, 93, 108] clientAgent=Prysm
[08:36:08] Error downloading in BlockInputSync.fetchBlockInput slot=6 ... peerClient=Prysm code=DOWNLOAD_BY_ROOT_ERROR_NO_SIDECAR_RECEIVED
           (repeats roughly every slot for slots 5..39, against both prysm peers)
[08:40:49] onMetadata peer=16...cLJokZ custodyGroupCount=128   <- refreshed, requests still empty
[08:42:44] initiating goodbyeAndDisconnect peer reason=Peer score too low, peerId=16...qm81eb

Possibly related (one-off, logged exactly once per Prysm node at startup, vs geth 1.17.6-unstable c6c7ffbb):

[08:35:56.08] ERROR sync: Failed to process sidecars from execution from block error=process data column
sidecars from execution: reconstruct data column sidecars: fetch cells and proofs from execution client
for block 0x7f15...c3b: get blobs V2/3: got an unexpected error in JSON-RPC response: invalid character 'm'
after object key:value pair root=0x7f15...c3b slot=4

Platform(s)

Linux (ARM) — docker (linux/arm64) on macOS host via Kurtosis

What version of Prysm are you running? (Which release)

Prysm/v7.1.8-6642080 (linux arm64)

Anything else relevant (validator index / public key)?

Peer clients: Lodestar v1.45.0 (65cfb73), EL: geth 1.17.6-unstable (c6c7ffbb).