fix(stream_cc3): progress watchdog on the finalized subscription; ws ping/pong on the cc3 client

[DylanVerstraete]

Second PR of the prover liveness audit, finding 2. Stacked on #1350; retarget to usc-dev once that merges.

Problem

A finalized-heads subscription can stay open and silent (dropped server-side, a proxy that stopped forwarding, a half-open peer). Both the seed and the steady-state loop in StreamCC3 awaited try_next() with no deadline, so the stream and every consumer (prover caches, attestor production) hung forever with no error to react to. cc_client::Client used RpcClient::from_insecure_url, which leaves jsonrpsee WebSocket pings off, so a half-open socket never surfaced a transport error either. The workspace reconnecting-rpc-client feature was inert: it is a different client type that was never instantiated.

Changes

• Progress watchdog in StreamCC3 (with_progress_timeout, default 90 s). On expiry the stream reads the node's finalized head point-to-point:
  • node ahead of us → the subscription is dead: reconnect, re-subscribe, the existing parent walk fills the gap
  • node not ahead → finality is stalled upstream: keep waiting, log at warn
  • probe fails → reconnect
    The seed applies the same deadline so construction cannot hang on a subscription that never delivers.
• stream::cc3::Progress (height, last-advance time, silent-recovery count) shared through the config. The prover publishes it on /health as cc3_finalized_height, cc3_finalized_age_seconds, cc3_silent_recoveries. This is the per-block progress signal the flattened event stream had lost.
• WebSocket ping/pong on the cc3 client (30 s interval, 45 s inactive limit, 1 failure). Mirrors what subxt builds (same per-subscription buffer) plus pings. Client::finalized_head_number() added for the probe.
• cc_client::ws_fixture (cargo feature ws-fixture, tests only): an in-process fake Creditcoin node with a controllable finalized head, per-connection subscription silencing, socket closing and pruned-state errors. The proof-gen tests from fix(proof-gen): share one Arc<CcClient> and supervise the cc3 event task #1350 now use it instead of a local copy.

The attestor consumes the same stream and inherits the watchdog with the default timeout; no attestor code changes.

Tests

common/streams/cc3/tests/watchdog.rs:

• silent subscription while the node keeps finalizing → replaced with exactly one redial, heights 2 and 3 arrive gap-free, silent_recoveries == 1
• stalled finality → no redial across several deadlines, resumes on the same socket
• a subscription that never delivers its first block → seed redials instead of hanging

cargo test -p stream_cc3 -p cc-client -p proof-gen-api-server green; clippy -D warnings on cc-client (both feature sets), stream_cc3, proof-gen-api-server, attestor; fmt, taplo, machete.

[cursor]

PR Summary

Medium Risk
Changes production Creditcoin RPC transport and finalized-block streaming recovery; mis-tuned timeouts or probe logic could cause extra reconnects or delayed detection, though behavior is covered by new fixture tests.

Overview
Fixes silent finalized-head subscriptions and half-open WebSocket cases that could leave StreamCC3 (and downstream prover/attestor consumers) hung with no error.

cc-client now builds the JSON-RPC WebSocket with ping/pong (30s / 45s inactive) instead of from_insecure_url, and adds finalized_head_number() for watchdog probes. A test-only ws-fixture feature exposes an in-process fake node (controllable finality, silent subscriptions, connection close, pruned storage); proof-gen’s duplicate fixture is removed in favor of this.

stream_cc3 adds a progress watchdog (default 90s on try_next, including stream seeding): if the node’s finalized head is ahead, it treats the subscription as dead, reconnects, and backfills; if finality is stalled it keeps waiting; probe failure triggers reconnect. Progress (height, last advance, silent-recovery count) is optional via config.

proof-gen-api-server shares that progress into the cc3 event task and /health (cc3_finalized_height, cc3_finalized_age_seconds, cc3_silent_recoveries). New watchdog.rs integration tests cover silent subscriptions, stalled finality, and seed hang avoidance.

^{Reviewed by Cursor Bugbot for commit 2395d7d. Bugbot is set up for automated code reviews on this repo. Configure here.}