fix(proof-gen): share one Arc<CcClient> and supervise the cc3 event task

[DylanVerstraete]

First PR of the prover liveness audit (2026-09-11), findings 7 and 1.

Problems

• Finding 7. CcClient::clone creates an independent connection slot (a fresh ArcSwap). Server value-cloned the client into each ContinuityBuilder and again into the event task. After a socket close the event task reconnected its copy; the builders stayed on the dead socket permanently (reproduced with a loopback WS node: the reconnected client succeeds, the builder-side client keeps failing).
• Finding 1. StreamCC3 ends itself only when history is permanently gone (pruned state, or a reconnect gap past MAX_BACKFILL_BLOCKS) and its contract is that the consumer restarts. Server::run spawned the event task and dropped the handle; the HTTP server kept serving from caches that no longer tracked the chain, so newer proofs failed with BlockNotReady and missed checkpoint/reversion events were never repaired.

Changes

• One Arc<CcClient> is created in Server::new and handed to every builder and to the event task. start_cc3_event_subscription takes Arc<CcClient>.
• Server::run supervises the event task through a new supervise() helper. When the task ends (error, clean return, or panic) the server marks the replica dead (ContinuityService::mark_event_stream_dead), asks axum to drain, waits up to 5 s for in-flight requests, and returns an error so the process exits nonzero and the orchestrator restarts it from a fresh finalized head. This is the same policy the attestor's production task applies to the same stream.
• /api/v1/health reports degraded with cc3_event_stream_dead: <reason> during the drain window. Readiness/503 semantics come in a later PR of this series (finding 5).

Tests

• 4 unit tests for supervise: event-task error, event-task panic, shutdown signal, HTTP server exit.
• tests/liveness_event_task.rs with a disposable in-process WebSocket "Creditcoin node" (tests/cc3_ws_fixture): reconnect through the shared Arc repairs every holder with exactly one redial; pruned history makes the event task return End of unbounded event stream.
• cargo test -p proof-gen-api-server (81 unit, 27 route, 2 liveness), clippy -D warnings, fmt, taplo, machete.

Series

A (this) → B finding 2 (stream progress watchdog) → C findings 3+4 (ETH repair lock, tip failover) → D findings 5+8 (readiness, consistent startup) → E finding 6 (admission budget, single-flight fills). Each stacks on the previous branch.

[cursor]

PR Summary

Medium Risk
Changes core runtime behavior: process now exits when the CC3 event stream ends, and all CC3 RPC paths share one client—correct for liveness but affects deployment restarts and connection lifecycle.

Overview
Fixes prover liveness by sharing one Arc<CcClient> across continuity builders and the CC3 event subscription, so a single reconnect() repairs every consumer instead of leaving builders on a dead socket after the event task reconnects its own clone.

Server::run no longer fire-and-forgets the event task: a new supervise() loop treats CC3 subscription exit (error, clean end, or panic) as fatal—calls ContinuityService::mark_event_stream_dead, drains HTTP for up to 5s, and returns Err so the process exits nonzero for orchestrator restart. Graceful shutdown still drains HTTP and returns Ok.

/api/v1/health reports degraded when the event stream is dead and exposes optional cc3_event_stream_dead with the reason during the drain window.

Adds supervise unit tests, a loopback cc3_ws_fixture WebSocket node, and liveness_event_task integration tests (shared-arc reconnect, pruned-history stream end).

^{Reviewed by Cursor Bugbot for commit f960ef5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[gluwa-bot]

Overview

Image reference	gluwa/creditcoin3:latest	gluwa/creditcoin3:latest
- digest	dfb918c3c546	c612eb51775f
- tag	latest	latest
- provenance	7e92a94	f44c175
- vulnerabilities
- platform	linux/amd64	linux/amd64
- size	430 MB	431 MB (+506 kB)
- packages	405	404 (-1)
Base Image	ubuntu:26.04 also known as: • latest • resolute	ubuntu:26.04 also known as: • latest • resolute • rolling
- vulnerabilities

Labels (1 changes)

• ± 1 changed
• 3 unchanged

-org.opencontainers.image.created=2026-06-27T04:19:04.617438+00:00
+org.opencontainers.image.created=2026-08-17T09:02:45.677319+00:00
 org.opencontainers.image.description=The Ubuntu container image maintained by Canonical

Ubuntu is a Debian-based Linux operating system that runs from the desktop to the cloud, to all your internet connected things.
It is the world's most popular operating system across public clouds and OpenStack clouds.
It is the number one platform for containers; from Docker to Kubernetes to LXD, Ubuntu can run your containers at scale.
Fast, secure and simple, Ubuntu powers millions of PCs worldwide.

 org.opencontainers.image.title=ubuntu
 org.opencontainers.image.version=26.04

Policies (2 improved, 0 worsened)

Policy Name	gluwa/creditcoin3:latest	gluwa/creditcoin3:latest	Change	Standing
Default non-root user	✅	✅		No Change
No copyleft licenses	⚠️ 373	⚠️ 361	-12	Improved
No fixable critical or high vulnerabilities	⚠️ 19	⚠️ 15	-4	Improved
No high-profile vulnerabilities	✅	✅		No Change
No outdated base images	⚠️	⚠️		No Change
No unapproved base images	✅	✅		No Change
Supply chain attestations	✅	✅		No Change

Packages and Vulnerabilities (47 package changes and 15 vulnerability changes)

• ➖ 1 packages removed
• ♾️ 46 packages changed
• 337 packages unchanged

• ✔️ 15 vulnerabilities removed

Changes for packages of type deb (35 changes)

	Package	Version gluwa/creditcoin3:latest	Version gluwa/creditcoin3:latest
♾️	base-files	14ubuntu6.1	14ubuntu6.2
♾️	bsdutils	1:2.41.3-3ubuntu2	1:2.41.3-3ubuntu2.2
♾️	curl	8.18.0-1ubuntu2.3	8.18.0-1ubuntu2.5
♾️	diffutils	1:3.12-1	1:3.12-1ubuntu0.1
♾️	gnu-coreutils	9.7-3ubuntu2	9.7-3ubuntu2.1
♾️	gpgv	2.4.8-4ubuntu3	2.4.8-4ubuntu3.1
♾️	libattr1	1:2.5.2-4	1:2.5.2-4ubuntu0.1
♾️	libaudit-common	1:4.1.2-1build1	1:4.1.2-1ubuntu0.1
♾️	libaudit1	1:4.1.2-1build1	1:4.1.2-1ubuntu0.1
♾️	libblkid1	2.41.3-3ubuntu2	2.41.3-3ubuntu2.2
♾️	libbz2-1.0	1.0.8-6build2	1.0.8-6ubuntu0.1
♾️	libc-bin	2.43-2ubuntu2.3	2.43-2ubuntu2.4
♾️	libc-gconv-modules-extra	2.43-2ubuntu2.3	2.43-2ubuntu2.4
♾️	libc6	2.43-2ubuntu2.3	2.43-2ubuntu2.4
♾️	libcurl4t64	8.18.0-1ubuntu2.3	8.18.0-1ubuntu2.5
♾️	libgcrypt20	1.12.0-2ubuntu1	1.12.0-2ubuntu1.1
♾️	libmount1	2.41.3-3ubuntu2	2.41.3-3ubuntu2.2
♾️	libpam-modules	1.7.0-5ubuntu3.1	1.7.0-5ubuntu3.2
♾️	libpam-modules-bin	1.7.0-5ubuntu3.1	1.7.0-5ubuntu3.2
♾️	libpam-runtime	1.7.0-5ubuntu3.1	1.7.0-5ubuntu3.2
♾️	libpam0g	1.7.0-5ubuntu3.1	1.7.0-5ubuntu3.2
♾️	libpq5	18.4-0ubuntu0.26.04.1	18.6-0ubuntu0.26.04.1
♾️	libsmartcols1	2.41.3-3ubuntu2	2.41.3-3ubuntu2.2
♾️	libssh2-1t64	1.11.1-1ubuntu0.26.04.3	1.11.1-1ubuntu0.26.04.4
♾️	libssl3t64	3.5.5-1ubuntu3.3	3.5.5-1ubuntu3.5
♾️	libsystemd0	259.5-0ubuntu3.3	259.5-0ubuntu3.4
♾️	libudev1	259.5-0ubuntu3.3	259.5-0ubuntu3.4
♾️	libuuid1	2.41.3-3ubuntu2	2.41.3-3ubuntu2.2
♾️	login	1:4.16.0-2+really2.41.3-3ubuntu2	1:4.16.0-2+really2.41.3-3ubuntu2.2
♾️	mount	2.41.3-3ubuntu2	2.41.3-3ubuntu2.2
♾️	openssl	3.5.5-1ubuntu3.3	3.5.5-1ubuntu3.5
♾️	openssl-provider-legacy	3.5.5-1ubuntu3.3	3.5.5-1ubuntu3.5
♾️	perl-base	5.40.1-7ubuntu0.1	5.40.1-7ubuntu0.3
♾️	util-linux	2.41.3-3ubuntu2	2.41.3-3ubuntu2.2
♾️	zlib1g	1:1.3.dfsg+really1.3.1-1ubuntu3	1:1.3.dfsg+really1.3.1-1ubuntu3.1

Changes for packages of type golang (6 changes)

	Package	Version gluwa/creditcoin3:latest	Version gluwa/creditcoin3:latest
♾️	github.com/canonical/pebble	1.31.1-0.20260528050051-33f10658d3fd	1.32.2-0.20260721212935-faa1696b477d
♾️	github.com/gorilla/websocket	1.5.1	1.5.3
		Removed vulnerabilities (1):
➖	golang.org/x/net	0.40.0
		Removed vulnerabilities (10):
♾️	golang.org/x/sys	0.33.0	0.46.0
		Removed vulnerabilities (1):
♾️	golang.org/x/term	0.32.0	0.44.0
♾️	stdlib	1.26.3	1.26.5
		Removed vulnerabilities (5):

Changes for packages of type npm (6 changes)

	Package	Version gluwa/creditcoin3:latest	Version gluwa/creditcoin3:latest
♾️	@types/node	26.1.2	22.7.5
♾️	node-gyp	13.0.1	13.0.2
♾️	picomatch	4.0.5	4.0.7
♾️	undici	8.10.0	8.10.2
♾️	undici-types	8.3.0	6.21.0
♾️	ws	8.21.2	8.21.3