fix(dkg): restart decrypt worker whenever an active round exists#855
Conversation
On a node restart, the TDH2 decrypt worker was only restarted for the latest round; if the next round had already opened registration, the latest round was no longer active and no start path fired, so decrypt requests queued indefinitely. ResumeDKGService now (re)starts the idempotent worker whenever an active DKG round exists, independent of the latest round's stage.
|
Verified the fix on a devnet: restarting The worker has no deadline anywhere downstream. Also, when there's no active round (worker not started, |
Add Debug logs at the two spots where the decrypt worker silently does nothing: when there is no active round so the worker is not started, and when the running flag CompareAndSwap no-ops because the worker is already running. Without these, a healthy-but-idle worker cannot be told apart from a stuck flag over a dead goroutine. Debug (not Info) since this runs every block.
The decrypt worker runs on a process-lifetime context.Background with no deadline, and it was passed straight into the kernel PartialDecryptTDH2 RPC. A wedged kernel (accepts the connection but never responds) blocked the worker's synchronous loop forever, so decryptWorkerRunning was never reset and ResumeDKGService's CompareAndSwap kept failing, leaving the worker dead with no self-heal. Wrap each kernel RPC the worker makes in a context.WithTimeout derived context (decryptKernelCallTimeout, 60s, matching the kernel-op budget in dkgAsyncTimeout and the contract-call timeout), and add gRPC keepalive to the kernel client dial options so a wedged connection is detected at the transport layer. Node-local worker path only; no consensus code touched.
07b3e2d to
0e512c6
Compare
Problem
The TDH2 decrypt worker was only (re)started for the latest round. If a node restarted while the next round had already opened registration (latest round != active round), no start path fired:
ResumeDKGService's recovery branch gates on the latest round's stage beingActive, which it isn't once the next round opened. Decrypt requests then queued indefinitely until the next round activated — days on Aeneid. This took Aeneid CDR decryption fully down after the gethv1.2.2upgrade restart.Fix
In
ResumeDKGService, (re)start the idempotent worker whenever an active DKG round exists, independent of the latest round's stage.LatestActiveRoundis written only on the finalize success path, so it stays intact across the next round's churn and across restarts.Tests
TestResumeDKGService_ActiveRoundStartsWorkerTestResumeDKGService_NoActiveRoundNoWorkerTestResumeDKGService_CompletedActiveStartsWorkerissue: #854