Skip to content

Fix parallel-call eval scoring, Muon update, and assorted correctness issues - #42

Open
Joe-Khawand wants to merge 1 commit into
cactus-compute:mainfrom
Joe-Khawand:fix/parallel-eval-and-correctness
Open

Fix parallel-call eval scoring, Muon update, and assorted correctness issues#42
Joe-Khawand wants to merge 1 commit into
cactus-compute:mainfrom
Joe-Khawand:fix/parallel-eval-and-correctness

Conversation

@Joe-Khawand

@Joe-Khawand Joe-Khawand commented Jul 23, 2026

Copy link
Copy Markdown

First time contributing here — spent some time in the training and inference paths and put together patches for a few things that looked like bugs. Happy to split them if that helps review.

Evaluation (training/eval.py)

  • Call P/R used set intersection, so identical parallel calls collapsed (emitting one of two identical required calls scored full recall). Now multiset-counted.
  • Arg/value metrics always compared against ref_by_name[name][0], mis-scoring parallel_multiple. Now greedy bipartite matching to distinct reference instances.
  • Added BFCL-style category buckets (simple / multiple / parallel / parallel_multiple) plus irrelevance, relevance and false-trigger rates. Scoring split into a pure score_tool_calls().

Training

  • Muon (optim.py): the raw gradient was orthogonalized before momentum, so the applied update was a sum of orthogonal matrices — not itself semi-orthogonal. Now momentum runs on the raw gradient and the Nesterov-blended update is orthogonalized; added max(1, fan_out/fan_in)^0.5 scaling. This shifts update magnitude, so muon_lr will want re-tuning.
  • z-loss (train.py, pretrain.py): averaged over padding while CE is masked, so its weight drifted with the padding fraction. Now masked to real tokens.

Inference (model/run.py, model/architecture.py)

  • decode() can project only the current position instead of the full (B, T, d) @ (d, V) buffer every step — numerically identical, training path unchanged. Projection only; a KV cache is the bigger win and left for a follow-up.
  • Streaming decoded each token in isolation, dropping SentencePiece spaces and splitting multibyte byte-fallback pieces. Now emits the decode delta.

Data (dataset/)

  • Loss mask marked only the first occurrence of each name/value, leaving repeats across parallel calls at base weight. Now marks all.
  • Boolean-polarity validator matched substrings ("on" inside "location", "song", …), which disabled the inverted-boolean filter. Now whole-word / phrase.

Validation

No access to the training data, so this is validated without a training run: the pure-Python logic (eval scoring, Muon ordering, polarity, loss mask) has unit tests, and the inference changes are checked against needle.pkl (decode(cur_pos=k) is bit-identical to the full projection; generation unchanged). The training effects of the Muon/z-loss and data changes aren't validated — they'll want an LR sweep / regeneration on your side.

Glad to take feedback and adjust anything — and thanks for open-sourcing this.

@Joe-Khawand
Joe-Khawand force-pushed the fix/parallel-eval-and-correctness branch 2 times, most recently from a467f9d to 0e28f5e Compare July 24, 2026 11:47
… issues

Evaluation (training/eval.py)
- Score calls as multisets so identical parallel calls are no longer
  collapsed by set-dedup (fixes call precision/recall on parallel).
- Match predicted args to distinct reference instances via greedy
  bipartite matching instead of always ref[0], fixing value_acc and the
  param metrics for parallel_multiple.
- Add BFCL-style category buckets (simple / multiple / parallel /
  parallel_multiple) plus irrelevance, relevance and false-trigger rates.
- Split the scoring out into a pure score_tool_calls() so it can be
  exercised without a checkpoint.

Training
- Muon: apply Nesterov momentum to the raw gradient and orthogonalize the
  blended update. The previous order orthogonalized first and ran momentum
  on the orthogonal factors, so the applied update was a sum of orthogonal
  matrices (not itself orthogonal). Also add aspect-ratio update scaling.
- Mask z-loss to non-pad positions in both train and pretrain; CE was
  already masked, so z-loss magnitude was scaling with the padding fraction.

Inference (model/run.py, model/architecture.py)
- decode() can project only the current position to the vocab, avoiding the
  full-buffer (B,T,d)@(d,V) matmul at every step. Numerically identical.
- Stream the decode delta of the whole sequence rather than decoding each
  token in isolation, which dropped SentencePiece spaces and split
  multibyte byte-fallback pieces.

Data (dataset/dataset.py, dataset/generate.py)
- Weight all repeated tool names / argument values in the loss mask, not
  just the first occurrence (matters for parallel calls).
- Match the boolean-polarity validator on whole words rather than
  substrings ("on" was matching inside "location", "song", ...), which had
  effectively disabled the inverted-boolean filter.

Signed-off-by: Joe Khawand <93840910+Joe-Khawand@users.noreply.github.com>
@Joe-Khawand
Joe-Khawand force-pushed the fix/parallel-eval-and-correctness branch from 0e28f5e to 9b22a8b Compare July 24, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant