Skip to content

📊 telemetry: cycle-consistent per-miner attribution for the dashboard - #197

Merged
george-connito merged 1 commit into
stagingfrom
feat/cycle-consistent-miner-telemetry
Jul 23, 2026
Merged

📊 telemetry: cycle-consistent per-miner attribution for the dashboard#197
george-connito merged 1 commit into
stagingfrom
feat/cycle-consistent-miner-telemetry

Conversation

@george-connito

Copy link
Copy Markdown
Collaborator

Additive only — no changes to evaluation, scoring, or weights. The subnet-api-gateway feature-detects these metrics, so deploy order is free in both directions.

New per-miner families (miner_uid-labeled, set from finalize):

  • validator_miner_last_scored_round_id — round_id of the last finalize verdict (scored / tie-zeroed / validation-failed / freeze-zero).
  • validator_miner_round_delta — raw (baseline − val_loss)^1.2 signal for evaluated uids; distinct from score_latest (finalized podium score) so the dashboard can show "how close was I" for non-podium miners.
  • validator_miner_evaluated_commit_info{hf_repo_id, hf_revision} — valued with the freeze round_id; strictly ≤1 labelset per uid (previous labelset evicted under a lock on commit change; KeyError-guarded for the post-restart case).

Emission moved INTO finalize_round_scores (after the journal flip): score snapshots (latest/avg/samples/emitted_at) now publish for EVERY verdict uid, not just weight recipients (~1 uid before), and the journal-recovery replay re-publishes everything after a restart — closing the "restart blanks the dashboard until the next full round" gap observed live during the 2026-07-23 cycle-api outage. The redundant snapshot loop in run.py's weight block is removed (WEIGHT_SUBMITTED stays).

RoundJournal bumped to v2: adds uid_to_commit (uid -> (hf_repo_id, hf_revision)) captured from uid_to_chain_checkpoint at freeze/finalize; v1 files still load (empty map); future versions rejected. _RecoveryRound hydrates checkpoint stubs so recovered finalizes emit commit info too.

Per-round series eviction: round_id label values emitted on validator_round_lifecycle_step / _miners_pending / _scored / _failed / bg_eval_lock_leak_total are tracked in-process and evicted via evict_round_series_before() on run.py's existing journal-prune cutoff — previously every round leaked permanent series.

Tests: 11 new (eviction invariant incl. restart simulation, journal v2/v1/future-version handling, finalize emission on live-shaped and recovery rounds, round-series eviction); full related suite (36) green in the staging image.

Additive only — no changes to evaluation, scoring, or weights. The
subnet-api-gateway feature-detects these metrics, so deploy order is
free in both directions.

New per-miner families (miner_uid-labeled, set from finalize):
- validator_miner_last_scored_round_id — round_id of the last finalize
  verdict (scored / tie-zeroed / validation-failed / freeze-zero).
- validator_miner_round_delta — raw (baseline − val_loss)^1.2 signal for
  evaluated uids; distinct from score_latest (finalized podium score) so
  the dashboard can show "how close was I" for non-podium miners.
- validator_miner_evaluated_commit_info{hf_repo_id, hf_revision} — valued
  with the freeze round_id; strictly ≤1 labelset per uid (previous
  labelset evicted under a lock on commit change; KeyError-guarded for
  the post-restart case).

Emission moved INTO finalize_round_scores (after the journal flip):
score snapshots (latest/avg/samples/emitted_at) now publish for EVERY
verdict uid, not just weight recipients (~1 uid before), and the
journal-recovery replay re-publishes everything after a restart —
closing the "restart blanks the dashboard until the next full round"
gap observed live during the 2026-07-23 cycle-api outage. The redundant
snapshot loop in run.py's weight block is removed (WEIGHT_SUBMITTED
stays).

RoundJournal bumped to v2: adds uid_to_commit (uid -> (hf_repo_id,
hf_revision)) captured from uid_to_chain_checkpoint at freeze/finalize;
v1 files still load (empty map); future versions rejected.
_RecoveryRound hydrates checkpoint stubs so recovered finalizes emit
commit info too.

Per-round series eviction: round_id label values emitted on
validator_round_lifecycle_step / _miners_pending / _scored / _failed /
bg_eval_lock_leak_total are tracked in-process and evicted via
evict_round_series_before() on run.py's existing journal-prune cutoff —
previously every round leaked permanent series.

Tests: 11 new (eviction invariant incl. restart simulation, journal
v2/v1/future-version handling, finalize emission on live-shaped and
recovery rounds, round-series eviction); full related suite (36) green
in the staging image.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@george-connito
george-connito merged commit 372b191 into staging Jul 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant