Internal metrics / introspection API

[michalkucharczyk]

Rationale

Today the only way to inspect smoldot's runtime state is via logs (system_health/system_peers aside, which expose almost nothing). Logs are often disabled in the field, verbose, and require reconstructing aggregate state by eye. This is especially painful in the browser, where you can't attach a debugger and bug reports come from users you can't instrument.

A small metrics surface lets us judge node state at a glance — peer churn, request failures, ban rate, sync progress, runtime-compilation cost — without enabling debug logging. Several useful internals are already measured and only log!ed (bytes/CPU, runtime compilation time, ping latency); this mostly aggregates what already exists.

Solution sketch

• A hand-rolled Arc<Metrics> of counters/gauges, no new dependencies, incremented at existing event sites.
• Surfaced over the existing JSON-RPC interface (works identically in wasm and full-node; no Prometheus HTTP server needed):
  • Pull: sudo_unstable_metrics() → snapshot of all metric families.
  • Push: sudo_unstable_metricsWatch(intervalMs) → periodic absolute snapshots (the client diffs for rates; this enables simple graphs in an enclosing app's debug window).
• Naming policy = Prometheus conventions (counter/gauge distinction, Total suffix, base units, dimensions as labels), camelCase to match the repo. No high-cardinality labels (no peerId/multiaddr/hash) — aggregate only; per-peer detail stays in system_peers.

Sample metrics (illustrative, not exhaustive)

• Network: networkConnectionsOpenedTotal{direction}, networkRequestsTotal{protocol,outcome}, networkPeerBansTotal{reason}, networkAddressesDroppedTotal{reason}, gauge networkGossipPeersConnected{chain}
• Sync: syncBlocksVerifiedTotal{outcome}, syncVerificationErrorsTotal{kind,reason}, gauges syncBestBlockHeight / syncFinalizedBlockHeight
• Runtime: runtimeCompilationSeconds, runtimeCacheHitsTotal (compilation timing already measured today)
• JSON-RPC / tx: jsonrpcRequestsTotal{method,outcome}, transactionsDroppedTotal{reason}
• Errors are first-class: classified by who's at fault (peer misbehavior / transient / local bug).

Usage scenarios

• Hidden debug view in the embedding app. A diagnostic screen subscribes via metricsWatch and renders live graphs (peer count, request failures, ban rate, sync height, runtime-compilation time) — turning "node feels stuck" into a readable picture without enabling trace logs.
• Centralized monitoring. An embedder/operator scrapes sudo_unstable_metrics and ships it to Grafana (full-node exposes it as a Prometheus /metrics endpoint later — a pure formatting layer over the same data), for fleet-wide dashboards and alerting.
• Large-scale / automated testing. Test harnesses assert on aggregate counters across many nodes — e.g. ban rate stays near zero, finality lag stays bounded, request-failure ratio under a threshold — to catch regressions like the recent UnknownTargetBlock false-ban, which today are only visible by eyeballing logs.

Scope / open questions

• v1: counters + gauges + already-measured durations (as count+sum). Full bucketed histograms deferred to v2.

[skunert]

In general I think this is a good idea. As discussed in meeting today, could be integrated in projects that rely on smoldot.

sudo_unstable_metricsWatch(intervalMs)

Regarding push based metrics, I am not so sure if we really need a subscription here. From my perspective subscription is mostly useful if smoldot sees some events that users are interested in, like a new block arriving. However in the metrics case, consumer knows best when it needs fresh metrics, so why bother with a subscription at all?

[michalkucharczyk]

fair point, not needed. Time based fetching can be on app side.

[lrubasze]

Related: #3301.
Both issues want introspection without parsing log text, and both expose it over JSON-RPC.

But they serve different use cases:

• Smoldot lifecycle event stream in light-base #3301 = UX, progress, status. Typed events the app shows to the user: first peer, warp progress, mode decision, bootstrap complete, stall. A dapp can render a real progress indicator and tell "no new blocks" apart from "disconnected". Client subscribes.
• Internal metrics / introspection API #3285 (this) = debugging, metrics, health. Counters and gauges for operators, dashboards, and tests: request failures, ban rate, compilation time, sync height. Answers "is the node healthy and why not". Client polls a snapshot.

Neither replaces the other: polled counters can't tell you the order of events or catch a short stall between two polls, and events can't give you failure ratios.

They can share:

• the instrumentation points in sync_service / network_service - one hook can update a counter and emit an event
• one error classification (Internal metrics / introspection API #3285's "who's at fault" maps to a stall reason here)
• naming and versioning rules for the unstable JSON-RPC surface