Skip to content

fix(dkg): restart decrypt worker whenever an active round exists#855

Merged
0xHansLee merged 3 commits into
dkg/devfrom
dkg/fix-decrypt-worker-resume
Jul 23, 2026
Merged

fix(dkg): restart decrypt worker whenever an active round exists#855
0xHansLee merged 3 commits into
dkg/devfrom
dkg/fix-decrypt-worker-resume

Conversation

@0xHansLee

Copy link
Copy Markdown
Contributor

Problem

The TDH2 decrypt worker was only (re)started for the latest round. If a node restarted while the next round had already opened registration (latest round != active round), no start path fired: ResumeDKGService's recovery branch gates on the latest round's stage being Active, which it isn't once the next round opened. Decrypt requests then queued indefinitely until the next round activated — days on Aeneid. This took Aeneid CDR decryption fully down after the geth v1.2.2 upgrade restart.

Fix

In ResumeDKGService, (re)start the idempotent worker whenever an active DKG round exists, independent of the latest round's stage. LatestActiveRound is written only on the finalize success path, so it stays intact across the next round's churn and across restarts.

Tests

  • TestResumeDKGService_ActiveRoundStartsWorker
  • TestResumeDKGService_NoActiveRoundNoWorker
  • TestResumeDKGService_CompletedActiveStartsWorker

issue: #854

On a node restart, the TDH2 decrypt worker was only restarted for the
latest round; if the next round had already opened registration, the
latest round was no longer active and no start path fired, so decrypt
requests queued indefinitely. ResumeDKGService now (re)starts the
idempotent worker whenever an active DKG round exists, independent of
the latest round's stage.
@yingyangxu2026

Copy link
Copy Markdown

Verified the fix on a devnet: restarting story in the window where the next round has opened registration while an active round still serves (latest != active) now brings the decrypt worker back up ("Decrypt worker started" + the worker picks up the active round's queue), which the old stage-gated path skipped. One remaining gap in the same failure family:

The worker has no deadline anywhere downstream. workerCtx is context.Background() (dkg_svc.go:295), passed straight into client.PartialDecryptTDH2 (:632), and kernel_client.go:59 dials with only transport creds — no per-call timeout, no keepalive. If the kernel wedges (accepts the connection but never responds), the worker's synchronous loop blocks indefinitely and never panics, so the decryptWorkerRunning flag is never reset and ResumeDKGService's CompareAndSwap keeps failing — the worker stays dead with no self-heal, the same end state as #854. Suggest a per-call timeout context plus gRPC keepalive dial params (and/or a watchdog that resets the flag on a stalled tick).

Also, when there's no active round (worker not started, :107) or the CompareAndSwap no-ops because the worker is already running (:289-291), nothing is logged, so a healthy-but-idle worker can't be told apart from a stuck flag over a dead goroutine. A Debug line at each would help (Info would be too noisy given it runs every block).

Add Debug logs at the two spots where the decrypt worker silently does
nothing: when there is no active round so the worker is not started, and
when the running flag CompareAndSwap no-ops because the worker is already
running. Without these, a healthy-but-idle worker cannot be told apart
from a stuck flag over a dead goroutine. Debug (not Info) since this runs
every block.
The decrypt worker runs on a process-lifetime context.Background with no
deadline, and it was passed straight into the kernel PartialDecryptTDH2
RPC. A wedged kernel (accepts the connection but never responds) blocked
the worker's synchronous loop forever, so decryptWorkerRunning was never
reset and ResumeDKGService's CompareAndSwap kept failing, leaving the
worker dead with no self-heal.

Wrap each kernel RPC the worker makes in a context.WithTimeout derived
context (decryptKernelCallTimeout, 60s, matching the kernel-op budget in
dkgAsyncTimeout and the contract-call timeout), and add gRPC keepalive to
the kernel client dial options so a wedged connection is detected at the
transport layer. Node-local worker path only; no consensus code touched.
@0xHansLee
0xHansLee force-pushed the dkg/fix-decrypt-worker-resume branch 2 times, most recently from 07b3e2d to 0e512c6 Compare July 23, 2026 05:26
@0xHansLee
0xHansLee merged commit 6c9e948 into dkg/dev Jul 23, 2026
7 checks passed
@0xHansLee
0xHansLee deleted the dkg/fix-decrypt-worker-resume branch July 23, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants