Staging - #196
Merged
Merged
Conversation
bittensor 10.1.0 can no longer query the live chain: the multi-mechanism runtime migration wrapped netuid storage keys in a NetUid newtype, so Subtensor.metagraph(netuid) dies with "Invalid type for data: 102 of type <class 'int'>" on both finney and archive. Validator bootstrap calls metagraph() (run._sync_lite_metagraph), so any :stable container restart crash-loops fleet-wide. Bump to bittensor 10.5.0 / bittensor-cli 9.23.2 and move the async-substrate-interface pin to the cyscale-based 2.2.1 (both new pins require it; the old scalecodec-era 1.6.4 hack is obsolete — btcli no longer pulls py-scale-codec). Verified in a clean python:3.10 container and layered over the :stable image on a live host (137.184.170.154): deps resolve with no scalecodec, all connito validator/shared modules import, metagraph returns n=256 on netuid 102, and the validator bootstraps through chain-commit fetch and model load. bittensor 11.0.0 was rejected: it removes Subtensor.metagraph and needs a real migration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y gates) Miners scored val_loss ≈ 0.04 vs baselines of 4.5–12.5 (perplexity ~1.04) with all miners within 0.0006 of each other. Root cause is the eval data layer, not the seed: the MinerCommit2 block-hash seed (PR #175/#177) was verified correct on chain for all completed rounds. Validated on the live validator by reproducing the round-8670446 eval stream with production code: - exp_legal still ran the legacy head-of-stream path (§2 shortcut of docs/exp-legal-migration-plan.md) — the whole draw came from the head of one sub-file; - 38% of streamed Multi_Legal_Pile rows have empty text → all-padding batches → NaN loss → silently excluded from the scored divisor; - 75% of non-empty rows share an identical 200-char prefix (templated boilerplate; zlib 0.364 vs c4 0.527); - truncation always scored each document's first 1024 tokens — its most templated region. Changes: - eval_shard_pick: row_count_source="verified_table" (frozen per-shard counts double as the shard allowlist), leaf_name_pattern override, and load_builder script-bypass (Multi_Legal_Pile's builder streams files from external repos and needs trust_remote_code; the generic json builder reads the pinned native shard directly). Register (joelniklaus/Multi_Legal_Pile, all_all) @ 911e1d21 with all 29 native shards above the 10k-row headroom floor (counted by full decompress). - dataloader: eval-only deterministic gates eval_min_text_chars=200 and eval_dedup_prefix_chars=200 (exact-prefix set — builtin hash() is per-process randomized and could break cross-validator consensus). - tokenize_windowed: long docs contribute a content-hash-derived window instead of the boilerplate prefix (no RNG, consensus-safe). Adopted by the default dataset class and exp_legal; exp_math left for follow-up. - exp_legal config: eval_source_seeded_shard_pick: true (shortcut retired). Miner training still streams the all_all builder mix, so eval ⊆ training distribution. - evaluate: scored_batches/nan_batches log promoted to INFO — the only production signal that an average is silently excluding NaN batches. Rollout: eval batches change; all validators must upgrade together (same discipline as the shuffle/skip bumps). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r 0.26) CI's clean-image build caught two conflicts the layered container test could not: bittensor 10.5.0 requires requests>=2.33 (repo pinned 2.32.3) and bittensor-cli 9.23.2 requires typer~=0.26 (repo pinned 0.24.1). Verified with a full pip dry-run resolve of requirements.txt in a clean python:3.10 container against the cu124 index. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…val-data' into staging
isabella618033
self-requested a review
July 21, 2026 21:23
isabella618033
approved these changes
Jul 21, 2026
Additive only — no changes to evaluation, scoring, or weights. The
subnet-api-gateway feature-detects these metrics, so deploy order is
free in both directions.
New per-miner families (miner_uid-labeled, set from finalize):
- validator_miner_last_scored_round_id — round_id of the last finalize
verdict (scored / tie-zeroed / validation-failed / freeze-zero).
- validator_miner_round_delta — raw (baseline − val_loss)^1.2 signal for
evaluated uids; distinct from score_latest (finalized podium score) so
the dashboard can show "how close was I" for non-podium miners.
- validator_miner_evaluated_commit_info{hf_repo_id, hf_revision} — valued
with the freeze round_id; strictly ≤1 labelset per uid (previous
labelset evicted under a lock on commit change; KeyError-guarded for
the post-restart case).
Emission moved INTO finalize_round_scores (after the journal flip):
score snapshots (latest/avg/samples/emitted_at) now publish for EVERY
verdict uid, not just weight recipients (~1 uid before), and the
journal-recovery replay re-publishes everything after a restart —
closing the "restart blanks the dashboard until the next full round"
gap observed live during the 2026-07-23 cycle-api outage. The redundant
snapshot loop in run.py's weight block is removed (WEIGHT_SUBMITTED
stays).
RoundJournal bumped to v2: adds uid_to_commit (uid -> (hf_repo_id,
hf_revision)) captured from uid_to_chain_checkpoint at freeze/finalize;
v1 files still load (empty map); future versions rejected.
_RecoveryRound hydrates checkpoint stubs so recovered finalizes emit
commit info too.
Per-round series eviction: round_id label values emitted on
validator_round_lifecycle_step / _miners_pending / _scored / _failed /
bg_eval_lock_leak_total are tracked in-process and evicted via
evict_round_series_before() on run.py's existing journal-prune cutoff —
previously every round leaked permanent series.
Tests: 11 new (eviction invariant incl. restart simulation, journal
v2/v1/future-version handling, finalize emission on live-shaped and
recovery rounds, round-series eviction); full related suite (36) green
in the staging image.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…elemetry 📊 telemetry: cycle-consistent per-miner attribution for the dashboard
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.