copyfree_battery_v8_tiny (7.1x cheaper eval) + torch-only imports + complete battery legs - #53
Open
sileod wants to merge 3 commits into
Open
copyfree_battery_v8_tiny (7.1x cheaper eval) + torch-only imports + complete battery legs#53sileod wants to merge 3 commits into
sileod wants to merge 3 commits into
Conversation
EvalLeg.keep selects examples by position AFTER the prefix limit, so a battery can be subsampled without touching the eval files. keep is omitted from the leg identifier when unset, so every pre-existing battery id is byte-identical (verified: v8 hashes the same before and after this change) and running shards are unaffected. copyfree_battery_v8_tiny: 6,504 of 46,427 scored sequences (7.1x), seeded-random indices (seed 0) from 809 arms of per-example sidecars. Reproduces the aux-task ranking at Spearman 0.9994 headline / 0.9994 cloze / 0.9949 pairwise. Random, not fitted anchors: OMP looked better under leave-one-shard-out but that split leaks (the same rgA tasks recur across shards) and it loses to random under a task-grouped split. Random also means the fidelity claim is about subset SIZE, so it survives any row reordering. Selection is by index, never by shrinking limit: limit is a prefix over unshuffled files and at b=16/f=10% it inflated the carry hop 30% (-0.0329 vs -0.0254). Also gitignore per_example/ -- sidecars are ~260KB/arm and origin is public. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0151Q74oFN2bRyuoXnTJLXir
load_battery_manifest defaults data_dir to data_cache -- which is what paper_battery and every influence run rely on -- but the test asserted the leg path resolved against the manifest own directory, so it could only ever pass by accident. Pass tmp_path explicitly, which is what the assertion was actually checking. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0151Q74oFN2bRyuoXnTJLXir
…ctually name transformers probes for tensorflow at import (image_transforms.py) whenever it can find it. This project is torch-only, so switch the probe off in the package inits and in conftest: it drops a multi-second import from every arm, and TF x86 aborts outright on hosts without SSE4.1. battery_legs.zip was missing 7 legs that the shipped manifests name, so a fresh clone could not resolve copyfree_battery_v8 at all -- and failed silently, since unpack only writes files that are absent. Added additively: every pre-existing entry is byte-identical, so no battery id moves.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent fixes to the eval battery, all verified against the 214 shipped cells.
1.
copyfree_battery_v8_tiny(7.1x) — a per-legkeepfield onEvalLegsubsets each leg's rows instead of dropping legs, so the compressed battery keeps all 39 legs. Leave-one-shard-out rank correlation with full v8 is rho 0.999; end-to-end arm time drops ~4.2x (eval is ~90% of an arm). Not the default:--eval-manifestselects it while v8 shards are in flight.2. Never import tensorflow — transformers probes for TF at import (
image_transforms.py) whenever it can find it. This project is torch-only, soUSE_TF=0/USE_FLAX=0in the package init and inconftest.py. It removes a multi-second import from every arm, and TF's x86 build aborts outright on hosts without SSE4.1 — which is why the test suite could not be collected at all on some machines.3.
battery_legs.zipwas missing 7 legs the shipped manifests name, socopyfree_battery_v8could not resolve from a fresh clone — silently, because unpack only writes files that are absent. Repaired additively: every pre-existing entry is byte-identical, so no battery id moves. Verified: v8 resolves from a clean tree with 0 missing legs at id1a482a2aeb5d, matching every existing cell.tests/test_training_runtime.pypasses 37/37.🤖 Generated with Claude Code
https://claude.ai/code/session_0151Q74oFN2bRyuoXnTJLXir