Sparse parity B6: Qwen3-8B + speed improvements - #7
Open
SilenNaihin wants to merge 10 commits into
Open
Conversation
- Model: Qwen3-14B → Qwen3-8B (~8B params, ~16GB bf16, fits in 80GB) - LR: 1e-6 → 5e-5 (50x increase, appropriate for LoRA) - max_parallel_calls: 1 → 4 (parallelize tool calls in rollout) - gpu_memory_utilization: 0.55 → 0.45 (avoid CUDA UVM spilling) - Updated critic model path and experiment name Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SilenNaihin
force-pushed
the
feat/sparse-parity-8b-speed
branch
from
April 10, 2026 16:28
5f5f217 to
4221d9f
Compare
VERL's vllm_async_server overwrites config max_model_len with the HF model's max_position_embeddings. Qwen3-8B has 40960 (vs 32768 for 14B). Actual sequence lengths are still bounded by max_prompt+max_response=32768. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hf_overrides rope_scaling was from 14B config and caused vLLM to derive max_position_embeddings=32768 (from original_max_position_embeddings), conflicting with the 40960 passed as max_model_len. Without the override, vLLM uses the model's native 40960 and our 32768 max_model_len fits fine. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Step 3 OOM: 97GB allocated by PyTorch on 80GB A100. vLLM 0.45 × 80 = 36GB leaves insufficient headroom for model weights + optimizer + activations. 0.35 × 80 = 28GB for vLLM, leaving ~52GB for training. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Still OOM at gpu_memory_utilization=0.35 — tried to allocate 14.38 GiB during backward pass. Gradient checkpointing recomputes activations instead of storing them, trading ~30% compute for significant memory savings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OOM at logsumexp: 40960 × 151936 vocab × 4 bytes = ~23 GB logits tensor. 16384 tokens → ~9.4 GB logits, fits within remaining GPU memory after vLLM (28 GB) + model weights (16 GB). Sequences longer than 16384 will be split across micro-batches automatically. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Step 4 (LR 3e-5) hit DMC 95,741 — big improvement from 637K baseline. Steps 5-6 (LR 4e-5, 5e-5) collapsed to 999M DMC with zero rewards. 3e-5 appears to be the sweet spot for LoRA r32 on 8B model. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…x_model_len
ppo_max_token_len_per_gpu resolves from ${max_model_len} in base config
and CLI overrides are silently ignored (Hydra structured config issue).
Reducing max_model_len from 32K to 16K halves the logits tensor:
16384 × 151936 × 4 = ~9.4 GB (was ~18.6 GB at 32K).
Context budget: 4K prompt + 12K response = 16K. Sparse parity solutions
are ~100 lines of numpy — 12K response is more than sufficient.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hydra structured config silently ignores YAML and CLI overrides for ppo_max_token_len_per_gpu (always resolves to 32768 from base config interpolation). Force it programmatically after config resolution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Older omegaconf version doesn't have read_write context manager. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
max_parallel_calls1 → 4 (rollout was 84% of step time)gpu_memory_utilizationfrom 0.55 to 0.45 (14B run oversubscribed at 113GB on 80GB A100)Expected improvements:
Test plan
sdporollout_probs_diff_meanis significantly > 0.005 after warmup🤖 Generated with Claude Code