Doppelganger check fails with 400 'No data submitted' when only not-recent keys are unknown to beacon state (REST mode)

[handelaar2]

Title

Doppelganger check fails fatally with 400: No data submitted when the only "not recent" keys are unknown to the beacon state (REST API mode)

---

Describe the bug

When running the validator client in REST API mode with --enable-doppelganger, the doppelganger check fails fatally and deterministically on every restart if the set of keys to be checked on-chain ("not recent" keys) consists entirely of pubkeys that are not yet present in the beacon state (e.g. deposits submitted but not yet processed — status UNKNOWN).

Root cause (in validator/client/beacon-api/doppelganger.go, v7.1.8):

1. Keys that signed within the last 2 epochs are classified as "recent" and excluded from the on-chain liveness check. In my setup, all active keys were attesting right up to the restart, so all of them are excluded.
2. The remaining "not recent" keys were the pubkeys with status UNKNOWN (deposits not yet processed). StateValidators() correctly returns zero matches for them, without error.
3. The indexes slice is therefore empty, but indexToLiveness() is still called unconditionally. The beacon node's /eth/v1/validator/liveness/{epoch} endpoint rejects an empty index list with 400: No data submitted, which the validator client treats as a fatal doppelganger failure.

The check then fails on every systemd restart in a loop, and the validator never starts. Note that the loop further down the function already handles "pubkey does not exist onchain" gracefully (continue when the pubkey has no index) — it is just never reached, because the liveness call fails first.

A guard skipping the liveness calls (or returning the "no doppelganger" response for those keys) when len(indexes) == 0 would fix this.

Has this worked before in a previous version?

Not tested in prior versions — observed on v7.1.8.

🔬 Minimal Reproduction

Setup:

• Single Prysm beacon node + validator client, both v7.1.8-51b5a75, mainnet
• Validator client in REST mode: --beacon-rest-api-provider=http://127.0.0.1:3500
• 7 keys in the wallet: 4 active and attesting, 3 with status UNKNOWN (deposited, not yet in beacon state)
• Beacon node fully synced (sync_distance: 0, not optimistic, EL online), uptime 10 days

Steps:

1. Ensure the active keys have attested within the last 2 epochs (normal operation).
2. Restart the validator client with --enable-doppelganger.
3. Doppelganger check fails fatally with the error below. Reproduces on every restart (3/3 consecutive systemd restarts in my case).

Verification that the beacon node endpoint itself is healthy:

$ curl -s -X POST http://localhost:3500/eth/v1/validator/liveness/467504 -H "Content-Type: application/json" -d '[]'
{"message":"No data submitted","code":400}

$ curl -s -X POST http://localhost:3500/eth/v1/validator/liveness/467504 -H "Content-Type: application/json" -d '["1000000"]'
{"data":[{"index":"1000000","is_live":false}]}

So the endpoint behaves to spec; the validator client is sending an empty index list.

Workarounds:

• Remove --enable-doppelganger (validator then starts and attests normally), or
• Wait until the pending validators are assigned indices (the state-validators lookup then returns matches and the check passes).

Error

INFO client: Running doppelganger check keyCount=7
ERROR client: Could not create validator runner error=could not succeed with doppelganger check: failed to get map from validator index to liveness for previous epoch 467502: failed to get liveness for epoch 467502: HTTP request unsuccessful (400: No data submitted)

Platform(s)

Linux (x86)

What version of Prysm are you running? (Which release)

v7.1.8 (Prysm/v7.1.8-51b5a75, built 2026-07-23), beacon node and validator client identical versions

Anything else relevant (validator index / public key)?

Possibly related to the error message in #15296 (same symptom string, different trigger — that issue concerned beacon node fallback and was fixed in #15401).