OutOfMemoryError

[squirrelfish]

model： nvidia/GLM-5.2-NVFP4
device： 8*A100

log：
Loading expert weights - 97.4% ..
Loading expert weights - 98.7% ..
Loading expert weights - 100.0% ..
Synchronizing with other peers..

CUDA graph initialized for bsz = 1: TPS = 56.176.
[rank6]: Traceback (most recent call last):
[rank6]: File "/opt/deepseek-tutel-accel/llm_moe_tutel.py", line 1292, in
[rank6]: build_cugraph()
[rank6]: File "/opt/deepseek-tutel-accel/llm_moe_tutel.py", line 797, in build_cugraph
[rank6]: step_once(bsz)
[rank6]: File "/opt/deepseek-tutel-accel/llm_moe_tutel.py", line 782, in
[rank6]: step_once = lambda bsz: forward_fn(batched_given_tokens[:bsz], batched_positions[max_concurrency:max_concurrency + bsz + 1], batched_pred_tokens[:bsz], max_pos=args.max_seq_len - 1)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/opt/deepseek-tutel-accel/./model_deepseek_moe.py", line 441, in forward_fn
[rank6]: splitData = torch.empty([chunk_size, _MAX_KV_SPLITS.value, Q.size(-2), 512], dtype=torch.float32, device=device)[:samples]
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU 6 has a total capacity of 79.33 GiB of which 405.81 MiB is free. Of the allocated memory 77.38 GiB is allocated by PyTorch
, with 1.02 GiB allocated in private pools (e.g., CUDA Graphs), and 763.27 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable
_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[ghostplant]

Will dive into this case soon. Is the A100 80G occupied by other services?

[squirrelfish]

Will dive into this case soon. Is the A100 80G occupied by other services?

No

[ghostplant]

It just consumes ~76GB per GPU on my environment, so any no issue found.
You can first try whether one of the following solutions could resolve your problem:

Option-1 (add CHUNK=1):

docker run -e CHUNK=1 ...

Option-2 (slightly reduce the max_seq_len):

docker run .. --max_seq_len 800000

[squirrelfish]

I restarted it and this time it worked. ~79GB per GPU on my environment

[squirrelfish]

“v1/messages“ api return empty object, "v1/chat/completions" is OK

[ghostplant]

Do you use v1/messages for claude code, or open-ai? They follow different API protocols. Tutel's v1/messages is Claude Code protocol.

[squirrelfish]

run the script:

import anthropic
from anthropic.types.message_param import MessageParam
client = anthropic.Anthropic(api_key='aaa',base_url="http://127.0.0.1:8000")
resp = client.messages.create(model='GLM-5.2', messages=[MessageParam(role='user', content='who are you')], max_tokens=1000)
print(resp.to_json())

it prints '{}'

[ghostplant]

run the script:

import anthropic
from anthropic.types.message_param import MessageParam
client = anthropic.Anthropic(api_key='aaa',base_url="http://127.0.0.1:8000")
resp = client.messages.create(model='GLM-5.2', messages=[MessageParam(role='user', content='who are you')], max_tokens=1000)
print(resp.to_json())
it prints '{}'

Thanks for pointing this out. Existing release bridges GLM with Claude Code CLI, VScode-Claude-Plugin and PowerShell listed in the README page. While cases beyond these scopes aren't yet covered at present, we may put it as a future TODO item.