expert_groups: add exp_legal expert group (MultiLegalPile) - #186
Merged
Conversation
present42
force-pushed
the
feat/exp-legal-expert-group
branch
from
July 3, 2026 18:09
b2825ce to
f7e9619
Compare
present42
added a commit
that referenced
this pull request
Jul 3, 2026
Flips TaskCfg.expert_group_name default from "exp_math" to "exp_legal"
and adds it to _LOCKED_FIELDS so operators can't drift back to the old
group after the subnet-wide switch. `auto_update_config` will reset any
non-default value on load and log a one-time reset warning; existing
operators' configs get updated in place on the first restart after the
new master lands.
Also updates the build_expert_assignment.py CLI defaults (--task and
--output) so someone running it fresh regenerates for exp_legal rather
than exp_math.
The default flip and the lock addition are both load-bearing:
* Default alone (no lock) lets operators override with exp_math and
fragments the network.
* Lock alone (no default flip) forces everyone to stay on exp_math —
the opposite of the intent.
Depends on #186 (which ships the expert_groups/exp_legal/ directory);
merge order must be #186 first, then this PR.
Verified:
* `TaskCfg()` resolves expert_group_name = "exp_legal", locked_defaults()
returns {'expert_group_name': 'exp_legal'}
* All 62 round/group tests still pass on top of #186.
Operator migration checklist for the activation date is separate and
lives in docs/exp-legal-activation-announcement.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
isabella618033
approved these changes
Jul 3, 2026
Scaffolds a new expert group for training on legal text, running in
parallel with the existing exp_math (group_id=0) and exp_dummy
(group_id=1). New chain expert_group token: 2.
Dataset: 50/50 interleave of joelniklaus/Multi_Legal_Pile (en_all
subset) and allenai/c4 (en) — mirrors exp_math's pattern of pairing the
specialized corpus with C4 for general-text grounding.
Two placeholder artifacts that must be replaced before launch (called
out in expert_groups/exp_legal/README.md):
* expert_assignment.json — verbatim copy of exp_math/'s. The MoE
router will load and not crash but routes legal tokens through
math-tuned experts. Regenerate via build_expert_assignment.py on a
machine with the model loaded.
* eval_source_seeded_shard_pick: false — opts out of the validator-
consensus seeded shard-pick path because (joelniklaus/Multi_Legal_Pile,
en_all) isn't registered in connito/shared/eval_shard_pick.py:
_KNOWN_SOURCES yet. Validators fall back to the legacy head-of-
stream eval path; consensus determinism is weaker than what exp_math
enjoys. Register a _SourceShardPolicy entry (with verified shard
rows) before flipping this back to the default.
No code changes elsewhere — the expert-group plumbing is already
dataset-agnostic at every component (miner, validator, sn_owner,
shared). Validators and miners opt into exp_legal by setting
task.expert_group_name in their config and restarting.
Verified loads cleanly:
* ExpertCfg.from_path(config.yaml) → group_id=2, two sources, 26
layers x 8 experts.
* ExpertManager(load_all_expert_groups=True) loads groups {0, 1, 2}
side-by-side without conflict.
* Existing group/round tests still pass (62/62 in
test_combined_validator_seed + test_round_freeze_groups +
test_round_groups).
Full design rationale, strategy, timeline, and pre-launch checklist:
docs/exp-legal-migration-plan.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…l_Pile
The shared dataloader's `_load_streaming_split` called HF's `load_dataset`
without ever passing `trust_remote_code`. Datasets that ship a custom
builder script (e.g. joelniklaus/Multi_Legal_Pile, whose `.py` loader
unpacks per-jurisdiction archives) refuse to load with:
ValueError: The repository for joelniklaus/Multi_Legal_Pile contains
custom code which must be executed... Please pass the argument
`trust_remote_code=True`.
This blocks both miner training and validator eval for any expert group
whose `dataset_sources` include such a dataset. exp_legal hits it on
every cycle right now (observed in the validator-george/main-hk2
container's logs at 2026-06-30 03:47:18; reported by the test-miner
copilot the same day).
Fix:
* Add `trust_remote_code: bool = False` to `DatasetSourceCfg`.
* Plumb it through `_load_streaming_split` and forward to
`load_dataset(trust_remote_code=True)` when set.
* Opt-in for `joelniklaus/Multi_Legal_Pile` in exp_legal/config.yaml;
`allenai/c4` stays off.
Per-source (not global) because trust_remote_code authorizes HF to
execute arbitrary Python from the dataset repo — operators consent one
dataset at a time. When seeded shard-pick gets wired up for
Multi_Legal_Pile (the deferred work in expert_groups/exp_legal/README.md
section 2), pair the trust flag with an `eval_source_revision_pin`
entry so the executed code is bound to a reviewed SHA.
Verified end-to-end against the live dataset: streaming load returns
rows with the expected `text` column populated (first row: 5892 chars
of an EU Official Journal extract). All round/group regression tests
still pass (62/62).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Anti-memorization: `en_all` totals ~36 k rows (uk_uk_lex 36,499 + switzerland_lexfind 147). At batch_size=4, seq_len=4096, 50/50 mix with c4, a miner burns through the full corpus ~3 times in ~15 hours of continuous training — well inside memorization-viable territory for DeepSeek-V2-Lite on structured legal text. Once one miner collapses to val_loss ≈ 0, ranking becomes noise-dominated (see PR #177's staging data where miners converged inside 1e-3 spread). `all_all` is the multilingual union — 33 shards, ~500 MB compressed, covering every language / document-type subset. Multi-order-of-magnitude larger eval pool. Cross-jurisdictional legal patterns encourage generalization rather than corpus-specific overfitting. DeepSeek-V2-Lite handles multilingual text well and the MoE router will distribute language-specific routing across experts naturally. Verified: builds live with trust_remote_code=True, streams multilingual rows (first samples land in Bulgarian caselaw via `joelito/eurlex_resources` — the builder's underlying source). Config still parses through ExpertCfg cleanly. Operator impact (called out in the migration doc): * Miners currently training against en_all should move existing globalver checkpoints aside and cold-start on all_all — the en_all-trained weights don't transfer well to a multilingual distribution * Baseline_loss will run higher on multilingual eval than on English-only; that's expected and doesn't affect rank ordering * The `expert_assignment.json` was first generated against en_all and should be regenerated against all_all before network activation to reflect multilingual routing patterns * README's `_KNOWN_SOURCES` follow-up now scopes to all 33 shards (not 2), and calls out the `_SHARD_NAME_FILTER` regex mismatch that also needs handling Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Flips TaskCfg.expert_group_name default from "exp_math" to "exp_legal"
and adds it to _LOCKED_FIELDS so operators can't drift back to the old
group after the subnet-wide switch. `auto_update_config` will reset any
non-default value on load and log a one-time reset warning; existing
operators' configs get updated in place on the first restart after the
new master lands.
Also updates the build_expert_assignment.py CLI defaults (--task and
--output) so someone running it fresh regenerates for exp_legal rather
than exp_math.
The default flip and the lock addition are both load-bearing:
* Default alone (no lock) lets operators override with exp_math and
fragments the network.
* Lock alone (no default flip) forces everyone to stay on exp_math —
the opposite of the intent.
Depends on #186 (which ships the expert_groups/exp_legal/ directory);
merge order must be #186 first, then this PR.
Verified:
* `TaskCfg()` resolves expert_group_name = "exp_legal", locked_defaults()
returns {'expert_group_name': 'exp_legal'}
* All 62 round/group tests still pass on top of #186.
Operator migration checklist for the activation date is separate and
lives in docs/exp-legal-activation-announcement.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Post-rebase architecture updates: - group_id 2 → 3: master assigned 2 to exp_c4_p02, the standing frozen helper slot that TaskCfg.helper_group_id=2 loads alongside every active task — exp_legal keeping 2 would collide in load_expert_group_assignment. Slot map is now 0=math, 1=dummy, 2=c4-helper, 3=legal; docs/README updated to match. - sequence_length 4096 → 1024: the fleet paradigm (locked DataCfg.sequence_length default). The lock machinery does not clamp per-group task configs, so 4096 would silently take effect and break the VRAM envelope miners/validators are sized for. - build_expert_assignment.py restored to master wholesale (the branch only tweaked argparse defaults; exp_legal is passed via --task at generation time instead). expert_assignment.json still carries the pre-tier4 uniform 8/layer placeholder — regenerated with natural routing in the next commit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The live tier4 assignments (exp_math 5-9 experts/layer, exp_c4_p02 8-12/layer — "_p02" = 2% share) were generated with a share-threshold method that never landed in-repo; the script only supported fixed top-N. Add --min-share: keep every expert whose share of the layer's router mass is >= the threshold (variable per-layer counts), with --experts-per-layer as the per-layer floor. Legacy fixed mode unchanged when the flag is omitted. Used to regenerate exp_legal's assignment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
present42
force-pushed
the
feat/exp-legal-expert-group
branch
from
July 11, 2026 03:52
60e5000 to
b26a95f
Compare
…share 0.035)
Replaces the pre-tier4 uniform 8/layer placeholder with a router-derived
assignment using the tier4 method: full DeepSeek-V2-Lite over 50 batches
of the actual exp_legal mix (Multi_Legal_Pile all_all 50% + c4-en 50%,
seq 1024), keeping experts with >= 3.5% of each layer's router-weight
mass (floor 5/layer).
The threshold was chosen by sweeping the recorded router stats for size
parity with exp_math, so miners carry a comparable expert load across
groups (same VRAM envelope, shard size, upload time):
exp_math : 187 pairs, 5-9 experts/layer (reference)
exp_legal @ 0.02 : 347 pairs, 8-18/layer (~1.8x — multilingual mix
spreads router mass more evenly)
exp_legal @ 0.035 : 179 pairs, 5-10/layer (chosen)
Generated on CPU inside the v0.2.20 validator container; loader gates
verified: groups {3,2} load together, my_ids unique per layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
present42
force-pushed
the
feat/exp-legal-expert-group
branch
from
July 11, 2026 11:10
b26a95f to
da924ac
Compare
…n bug) Found live on the pioneer validator (2026-07-11 11:49 UTC) during the exp_legal PR test: a config YAML still saying exp_math was auto-reset to the locked default exp_legal and PERSISTED to disk — but the process kept running exp_math (task_path=/app/expert_groups/exp_math, chain commits group_id:0), because task.path/task.exp are derived at config construction, before check_and_prompt_locked runs. Without this fix, every fleet validator would silently stay on the old group for one full restart cycle while claiming the new one — the activation would need two coordinated restarts instead of one. from_path now detects a locked-reset change of task.expert_group_name and re-derives the task config in the same load. The name is passed explicitly because the no-arg _update_by_task reloads task.exp from the stale task.path before refreshing paths. Regression test reproduces the exact scenario (exp_math YAML → auto_update → asserts group_id 3 + exp_legal path + persisted YAML all agree in one load). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hazard Two lessons from the live pioneer test (2026-07-11): resuming exp_math globalver checkpoints into the group-3 model contaminates it (bump run_name instead — checkpoint_path is derived and reverts manual edits), and miners switching before the validator stake-majority get deregistered (uid-121 test miner was pruned exactly this way). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Derive checkpoint_path as base/<coldkey>/<hotkey>/<run_name>/<expert_group_name> so switching the active expert group automatically writes to and resumes from a fresh, group-isolated directory — no run_name bump, and no risk of resuming another group's incompatible checkpoints (the exp_math->exp_legal overlay incident on 2026-07-11). Path is re-derived after the locked-field reset via _update_by_task -> _refresh_paths, so it always tracks the effective group. Update the migration plan (no run_name bump needed) and add a checkpoint_path assertion to the locked-task-rederive regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scaffolds a new expert group for training on legal text. New chain expert_group token: 2.
Dataset: 50/50 interleave of joelniklaus/Multi_Legal_Pile (en_all subset) and allenai/c4 (en) — mirrors exp_math's pattern of pairing the specialized corpus with C4 for general-text grounding.
Two placeholder artifacts that must be replaced before launch:
expert_assignment.json — verbatim copy of exp_math/'s. The MoE router will load and not crash but routes legal tokens through math-tuned experts. Regenerate via build_expert_assignment.py on a machine with the model loaded.
eval_source_seeded_shard_pick: false — opts out of the validator- consensus seeded shard-pick path because (joelniklaus/Multi_Legal_Pile, en_all) isn't registered in connito/shared/eval_shard_pick.py: _KNOWN_SOURCES yet. Validators fall back to the legacy head-of- stream eval path; consensus determinism is weaker than what exp_math enjoys. Register a _SourceShardPolicy entry (with verified shard rows) before flipping this back to the default.
No code changes elsewhere — the expert-group plumbing is already dataset-agnostic at every component (miner, validator, sn_owner, shared). Validators and miners opt into exp_legal by setting task.expert_group_name in their config and restarting.
Verified loads cleanly: