[Feature]: B300 (Blackwell, sm_103) support for the SLIME GRPO test case

[littlemex]

Update: implemented and open for review

Since filing this, I implemented every fix below, verified them end-to-end on both H200 and B300, and opened the PRs. This issue now tracks a completed, reviewable set rather than a proposal. I would appreciate a review of the epic PR (#1164) and its focused sub-PRs.

The SLIME GRPO test case (3.test_cases/pytorch/slime, added in #1129) now runs the full GRPO loop (rollout → train → weight sync → eval) for both Qwen3-4B (dense, colocated) and Qwen3-30B-A3B (MoE, disaggregated) on the CUDA-13 base image, after a small set of focused fixes. All fixes are landed on the epic branch behind #1164:

• Enable the SLIME GRPO test case to run end-to-end on B300 / H200 #1164 — [Don't merge] epic / integration PR that tracks the whole set.
• Fix numpy version in ddp test case requirements #2 — --sglang-log-level WARN → lowercase warning (uvicorn startup).
• Update README.md #3 — cu13 torch_memory_saver .so selection (self-neutralizing build-time patch).
• Update versions for Megatron-LM #4 — pin numpy<2 (Megatron init vs numpy 2.x).
• Add more information on placement group usage #5 — enable the 30B MoE disaggregated recipe (mbridge, GPU-less-driver validate guard, drop dead --sglang-enable-ep-moe + set CUDA_DEVICE_MAX_CONNECTIONS=1).

(PRs #2–#5 are on my fork against the epic branch; #1164 is the integration PR here.)

Two corrections vs my earlier description, from further verification: I dropped the large-HBM --sglang-cuda-graph-max-bs tuning (I could not reproduce a capture slowdown on H200, so I removed it as unverified rather than ship a preventive flag), and the "30B MoE online weight sync returns 400" item was wrong — no 400 occurs on SGLang's defaults; --sglang-enable-ep-moe turned out to be a dead flag that SGLang 0.5.12 silently ignores, so it is simply dropped. The real 30B blocker was elsewhere (Megatron validate_args probing the GPU on the GPU-less Ray driver), captured below.

Feature Description & Motivation

I used this test case as my reference to bring SLIME up on NVIDIA B300 (p6-b300, sm_103) and H200 (p5en) on EKS. The README documents only ml.p5.48xlarge (H100); there is no B300/p6 (or H200) coverage. Since the base image is already nvcr.io/nvidia/pytorch:26.02-py3 + SGLang 0.5.12.post1 + torch 2.11 / CUDA 13 (Blackwell-capable), the delta to add B300/H200 is small — almost entirely env/flag values plus a handful of fixes that are general (not GPU-generation-specific) and reproduce on the documented p5/H100 target too.

B300 is a natural gap here, and the repo already merges B300 work actively (#1070 sm_103, #1116 256×B300, #1142 EKS B300 EP-comparison), so there is a clear place for it. The findings are classified by root cause so each PR stays focused.

What was fixed (all implemented, verified, and linked to their PR)

General fixes — not B300-specific; they affect the documented p5/HyperPod path too (CUDA 13 / dependency issues):

1. --sglang-log-level WARN breaks uvicorn startup. Both recipe/run_grpo_qwen3_4b.sh and run_grpo_qwen3_30b_a3b.sh pass uppercase WARN. SGLang forwards it to uvicorn's log_level, which accepts lowercase only (critical/error/warning/info/debug/trace); 'WARN' raises KeyError and uvicorn dies before the HTTP server binds, so the rollout health check hangs forever. GPU-independent. Fix: lowercase warning (preserves the intended verbosity). → Fix numpy version in ddp test case requirements #2

2. The train worker dies on CUDA 13 from a cu12-linked torch_memory_saver .so. With offload on (the reference default), SLIME LD_PRELOADs torch_memory_saver_hook_mode_preload.abi3.so, which is cu12-linked (ldd → libcudart.so.12 => not found) on the CUDA-13 image, so the worker dies at startup. This reproduces on any CUDA-13 image (the test case's NGC 26.02 base), on H100/H200/B300 alike. Proper fix filed upstream in SLIME (delegate .so selection to torch_memory_saver's own CUDA-aware resolver); the test case carries the same fix in place via a self-neutralizing build-time patch (no fork, no forked URL, no-op once upstream merges it). → Update README.md #3

3. Megatron aborts on numpy 2.x. sglang[all] pulls numpy 2.x transitively, but Megatron hard-asserts numpy 1.x at init, and the Dockerfile does not pin numpy. numpy is pure-CPU, so this is GPU-generation- and CUDA-independent. Fix: pin numpy<2 as the last dependency step, matching upstream SLIME's own Dockerfile. → Update versions for Megatron-LM #4

4. MODEL_ARGS reaches train.py as zero elements, so training never starts. The recipes build the invocation into a string and submit it as ray job submit ... -- bash -c "${TRAIN_CMD}" with the model config deferred as \${MODEL_ARGS[@]}. With Ray pinned to ray[default]==2.55.1, ray job submit re-joins the post--- tokens with subprocess.list2cmdline and runs them via Popen(shell=True) → an outer /bin/sh -c, which expands ${MODEL_ARGS[@]} (a bash array from source scripts/models/*.sh) before the inner bash -c sources it, so it expands to 0 elements and train.py dies at hf_validate_args (hidden_size None). I confirmed this is not plain-EKS-specific with a slime-independent repro (single bash -c → 25 args; ray job submit -- bash -c → 0 args). Fix: expand ${MODEL_ARGS[@]} in the same shell that sources it and pass argv after -- (a small launcher script), matching SLIME's own run-qwen3-4B.sh. → Enable the SLIME GRPO test case to run end-to-end on B300 / H200 #1164 (first fix; detail comment on the PR)

30B MoE recipe only — not exercised by the Qwen3-4B path; needed only for run_grpo_qwen3_30b_a3b.sh. These are bundled into one experiment-scoped PR because the recipe does not reach training until all are in place:

5. 30B MoE torch_dist conversion needs mbridge. tools/convert_hf_to_torch_dist.py requires from mbridge import AutoBridge, which the --no-deps slime install does not pull (dense models ship a pre-converted torch_dist, so they never hit this). Fix: pip install --no-deps mbridge. → Add more information on placement group usage #5

6. Megatron validate_args probes the GPU on the GPU-less Ray driver. SLIME runs parse_args()/validate_args on the Ray driver, which is intentionally GPU-less in this test case (the head is num-gpus: 0). Megatron's validate_args eagerly probes the CUDA device — get_device_capability() for --moe-grouped-gemm and get_device_arch_version() for the TP/CP CUDA_DEVICE_MAX_CONNECTIONS note — so on the driver it raises RuntimeError: Found no NVIDIA driver. Only the MoE + TP/CP>1 recipe reaches these probes, which is why the 4B path never hits it. Fix: a self-neutralizing build-time patch guards both probes in SLIME's validate_args wrapper only when torch.cuda.is_available() is False (the driver); the real GPU actors are unchanged. The permanent fix belongs upstream in Megatron (guard the probes with torch.cuda.is_available()), and I intend to file it there. → Add more information on placement group usage #5

7. --sglang-enable-ep-moe is a dead flag; CUDA_DEVICE_MAX_CONNECTIONS=1 is missing. The 30B recipe passes --sglang-enable-ep-moe, which SGLang 0.5.12 removed from ServerArgs; SLIME parses --sglang-* leniently (parse_known_args / ignore_unknown_args), so it is silently ignored, not rejected — it configures nothing. And the recipe omits CUDA_DEVICE_MAX_CONNECTIONS=1, which Megatron asserts for TP/CP>1. Fix: drop the dead flag (SGLang's defaults serve the Qwen3-30B-A3B MoE correctly — moe_runner_backend=auto resolves to the triton runner for bf16, ep_size defaults to 1 — verified end-to-end with no MoE flag set, so no replacement is needed) and add CUDA_DEVICE_MAX_CONNECTIONS=1 to the recipe runtime-env. → Add more information on placement group usage #5

Verification (end-to-end, on hardware)

Validated on EKS with 2× p5en.48xlarge (16× H200, CUDA 13.0), and the general fixes additionally on 2× p6-b300.48xlarge (B300, sm_103). Same base image, Ray 2.55.1, SGLang 0.5.12.post1, torch 2.11, CUDA 13, SLIME v0.2.4 as the reference.

• Qwen3-4B, colocated, in-process reward (recipe/run_grpo_qwen3_4b.sh): completed 3 full GRPO loops (rollout → ref/actor log-probs → Megatron train step → weight sync back to SGLang), no regressions (libcudart.so.12, numpy-2.x assert, uvicorn KeyError, hidden_size None all absent). Inter-node NCCL used EFA (NET/OFI Selected provider is efa, fabric is efa-direct (found 16 nics) across all 16 ranks).
• remote_rm HTTP reward service (reward_service/, kubernetes/reward-service.yaml): built the CPU-only reward image, ran the math_verify backend on a non-GPU node, re-ran the 4B recipe with RM_TYPE=remote_rm. The rollout reached the service over HTTP: 250+ POST /score, all 200 OK, zero non-200, zero remote_rm failures on the SLIME side.
• Qwen3-30B-A3B, MoE, disaggregated (recipe/run_grpo_qwen3_30b_a3b.sh; 12 train GPUs at TP=2/EP=2/CP=2, 4 rollout GPUs): reached a full GRPO loop — Final collected 96 samples from rollout to train → Timer train start → ref_log_probs (27.4s) → actor_train (55.4s) → Timer train end (93.2s) → Timer update_weights end (12.1s) — with zero occurrences of Found no NVIDIA driver, not divisible by micro batch size, libcudart.so.12, or 400 Bad Request (the MoE online weight sync succeeded on SGLang's defaults).

Category

Enhancement to existing test case

Alternatives Considered

• Add a separate b300/ subdirectory (like the H200-vs-B300 approach in feat: add Qwen3-8B pre-training sample (H200 vs B300, NeMo/Megatron) #1138). I lean against this for SLIME because the required deltas are almost entirely env/flag values, and duplicating recipes would create the per-generation divergence the feat: add Qwen3-8B pre-training sample (H200 vs B300, NeMo/Megatron) #1138 review pushed back on. A single recipe with B300 as env/flag deltas plus a README section is preferable.
• Bundle everything into one "B300 support" PR. Avoided: it would mix the general bug fixes with the 30B-MoE-only fixes, making review hard. Hence the focused split (one epic PR tracking small sub-PRs).
• Downgrade SGLang to avoid MoE issues. Rejected: SGLang 0.5.9 forces torch 2.9.1, breaking the NGC torch-2.11/CUDA-13 ABI the base image relies on. SGLang 0.5.12 defaults serve the MoE correctly without a downgrade.
• Escape the array harder instead of using a launcher (item 4). Rejected: list2cmdline always double-quotes the payload, so the outer /bin/sh always expands ${MODEL_ARGS[@]} regardless of recipe-side quoting. The launcher (argv after --, no array crossing the ray boundary) is the only robust fix and matches SLIME upstream.

Additional Context

Environment: EKS (plain-EKS and SageMaker HyperPod EKS are the same family at this test-case level), 2× p6-b300.48xlarge (8× B300, sm_103, ~192 GB/GPU) and 2× p5en.48xlarge (8× H200), with EFA, FSx Lustre and KubeRay.

Questions for the maintainers:

1. Is the focused-PR split (one [Don't merge] epic PR tracking small sub-PRs) the shape you prefer, or would you rather each fix be its own separate PR against main?
2. For B300/H200, are env/flag deltas in the existing recipes preferred over a b300/ subdir, given the feat: add Qwen3-8B pre-training sample (H200 vs B300, NeMo/Megatron) #1138 divergence discussion?
3. For the cu13 torch_memory_saver .so, is a thin test-case-side self-neutralizing fallback acceptable until the CUDA-major selection lands upstream in SLIME?

[littlemex]

• No difference hyperpod eks and eks in this sample.
• H200 verification. cu12 to cu13

[allela-roy]

Thanks for filing issue @littlemex . ACK, will work on B300 support.

[littlemex]

Quick update: I've now implemented and verified end-to-end all the fixes discussed above, and opened them for review.
I've also updated the issue description to reflect the final, verified state (a couple of my earlier guesses didn't hold up under further testing and were dropped — details are in the updated description and the per-PR comments).

I'd really appreciate a review of the epic PR (#1164) and its sub-PRs whenever you have a chance. Happy to split, reshape, or rebase the set however works best for you.

[littlemex]

Upstream Issues/PRs

• SGLang: [Bug] Uppercase --log-level (e.g. WARN) hangs the HTTP server: unnormalized value forwarded to uvicorn raises KeyError before the socket binds sgl-project/sglang#30353
• Slime: [Bug] offload_train train worker dies on CUDA 13 with libcudart.so.12: cannot open shared object file — the torch_memory_saver LD_PRELOAD .so is chosen by filename, not by CUDA runtime THUDM/slime#2186
• Megatron-LM: [Bug] validate_args() eagerly probes the CUDA device and crashes GPU-less launcher/driver processes (Found no NVIDIA driver) NVIDIA/Megatron-LM#5763

[littlemex]

Closing this issue as the PR has been successfully merged.