📊 telemetry: cycle-consistent per-miner attribution for the dashboard - #197
Merged
Merged
Conversation
Additive only — no changes to evaluation, scoring, or weights. The
subnet-api-gateway feature-detects these metrics, so deploy order is
free in both directions.
New per-miner families (miner_uid-labeled, set from finalize):
- validator_miner_last_scored_round_id — round_id of the last finalize
verdict (scored / tie-zeroed / validation-failed / freeze-zero).
- validator_miner_round_delta — raw (baseline − val_loss)^1.2 signal for
evaluated uids; distinct from score_latest (finalized podium score) so
the dashboard can show "how close was I" for non-podium miners.
- validator_miner_evaluated_commit_info{hf_repo_id, hf_revision} — valued
with the freeze round_id; strictly ≤1 labelset per uid (previous
labelset evicted under a lock on commit change; KeyError-guarded for
the post-restart case).
Emission moved INTO finalize_round_scores (after the journal flip):
score snapshots (latest/avg/samples/emitted_at) now publish for EVERY
verdict uid, not just weight recipients (~1 uid before), and the
journal-recovery replay re-publishes everything after a restart —
closing the "restart blanks the dashboard until the next full round"
gap observed live during the 2026-07-23 cycle-api outage. The redundant
snapshot loop in run.py's weight block is removed (WEIGHT_SUBMITTED
stays).
RoundJournal bumped to v2: adds uid_to_commit (uid -> (hf_repo_id,
hf_revision)) captured from uid_to_chain_checkpoint at freeze/finalize;
v1 files still load (empty map); future versions rejected.
_RecoveryRound hydrates checkpoint stubs so recovered finalizes emit
commit info too.
Per-round series eviction: round_id label values emitted on
validator_round_lifecycle_step / _miners_pending / _scored / _failed /
bg_eval_lock_leak_total are tracked in-process and evicted via
evict_round_series_before() on run.py's existing journal-prune cutoff —
previously every round leaked permanent series.
Tests: 11 new (eviction invariant incl. restart simulation, journal
v2/v1/future-version handling, finalize emission on live-shaped and
recovery rounds, round-series eviction); full related suite (36) green
in the staging image.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Additive only — no changes to evaluation, scoring, or weights. The subnet-api-gateway feature-detects these metrics, so deploy order is free in both directions.
New per-miner families (miner_uid-labeled, set from finalize):
Emission moved INTO finalize_round_scores (after the journal flip): score snapshots (latest/avg/samples/emitted_at) now publish for EVERY verdict uid, not just weight recipients (~1 uid before), and the journal-recovery replay re-publishes everything after a restart — closing the "restart blanks the dashboard until the next full round" gap observed live during the 2026-07-23 cycle-api outage. The redundant snapshot loop in run.py's weight block is removed (WEIGHT_SUBMITTED stays).
RoundJournal bumped to v2: adds uid_to_commit (uid -> (hf_repo_id, hf_revision)) captured from uid_to_chain_checkpoint at freeze/finalize; v1 files still load (empty map); future versions rejected. _RecoveryRound hydrates checkpoint stubs so recovered finalizes emit commit info too.
Per-round series eviction: round_id label values emitted on validator_round_lifecycle_step / _miners_pending / _scored / _failed / bg_eval_lock_leak_total are tracked in-process and evicted via evict_round_series_before() on run.py's existing journal-prune cutoff — previously every round leaked permanent series.
Tests: 11 new (eviction invariant incl. restart simulation, journal v2/v1/future-version handling, finalize emission on live-shaped and recovery rounds, round-series eviction); full related suite (36) green in the staging image.