Fix parallel-call eval scoring, Muon update, and assorted correctness issues - #42
Open
Joe-Khawand wants to merge 1 commit into
Open
Conversation
Joe-Khawand
force-pushed
the
fix/parallel-eval-and-correctness
branch
2 times, most recently
from
July 24, 2026 11:47
a467f9d to
0e28f5e
Compare
… issues
Evaluation (training/eval.py)
- Score calls as multisets so identical parallel calls are no longer
collapsed by set-dedup (fixes call precision/recall on parallel).
- Match predicted args to distinct reference instances via greedy
bipartite matching instead of always ref[0], fixing value_acc and the
param metrics for parallel_multiple.
- Add BFCL-style category buckets (simple / multiple / parallel /
parallel_multiple) plus irrelevance, relevance and false-trigger rates.
- Split the scoring out into a pure score_tool_calls() so it can be
exercised without a checkpoint.
Training
- Muon: apply Nesterov momentum to the raw gradient and orthogonalize the
blended update. The previous order orthogonalized first and ran momentum
on the orthogonal factors, so the applied update was a sum of orthogonal
matrices (not itself orthogonal). Also add aspect-ratio update scaling.
- Mask z-loss to non-pad positions in both train and pretrain; CE was
already masked, so z-loss magnitude was scaling with the padding fraction.
Inference (model/run.py, model/architecture.py)
- decode() can project only the current position to the vocab, avoiding the
full-buffer (B,T,d)@(d,V) matmul at every step. Numerically identical.
- Stream the decode delta of the whole sequence rather than decoding each
token in isolation, which dropped SentencePiece spaces and split
multibyte byte-fallback pieces.
Data (dataset/dataset.py, dataset/generate.py)
- Weight all repeated tool names / argument values in the loss mask, not
just the first occurrence (matters for parallel calls).
- Match the boolean-polarity validator on whole words rather than
substrings ("on" was matching inside "location", "song", ...), which had
effectively disabled the inverted-boolean filter.
Signed-off-by: Joe Khawand <93840910+Joe-Khawand@users.noreply.github.com>
Joe-Khawand
force-pushed
the
fix/parallel-eval-and-correctness
branch
from
July 24, 2026 11:50
0e28f5e to
9b22a8b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First time contributing here — spent some time in the training and inference paths and put together patches for a few things that looked like bugs. Happy to split them if that helps review.
Evaluation (
training/eval.py)ref_by_name[name][0], mis-scoringparallel_multiple. Now greedy bipartite matching to distinct reference instances.score_tool_calls().Training
optim.py): the raw gradient was orthogonalized before momentum, so the applied update was a sum of orthogonal matrices — not itself semi-orthogonal. Now momentum runs on the raw gradient and the Nesterov-blended update is orthogonalized; addedmax(1, fan_out/fan_in)^0.5scaling. This shifts update magnitude, somuon_lrwill want re-tuning.train.py,pretrain.py): averaged over padding while CE is masked, so its weight drifted with the padding fraction. Now masked to real tokens.Inference (
model/run.py,model/architecture.py)decode()can project only the current position instead of the full(B, T, d) @ (d, V)buffer every step — numerically identical, training path unchanged. Projection only; a KV cache is the bigger win and left for a follow-up.Data (
dataset/)"on"inside "location", "song", …), which disabled the inverted-boolean filter. Now whole-word / phrase.Validation
No access to the training data, so this is validated without a training run: the pure-Python logic (eval scoring, Muon ordering, polarity, loss mask) has unit tests, and the inference changes are checked against
needle.pkl(decode(cur_pos=k)is bit-identical to the full projection; generation unchanged). The training effects of the Muon/z-loss and data changes aren't validated — they'll want an LR sweep / regeneration on your side.Glad to take feedback and adjust anything — and thanks for open-sourcing this.