'Attention' object has no attribute 'q_scale'.

[kisseternity]

Hello, when I use this project with verl, some error happens when loading model weights.
Model info: Qwen2.5-0.5B-Instruct
Error:
WorkerDict object at 0x7f648412eb60>)
File "/root/epfs/code/verl-self-dev-flash_attn3/verl/single_controller/ray/base.py", line 720, in func
return getattr(self.worker_dict[key], name)(*args, **kwargs)
File "/root/epfs/code/verl-self-dev-flash_attn3/verl/single_controller/base/decorator.py", line 514, in inner
return func(*args, **kwargs)
File "/root/epfs/code/verl-self-dev-flash_attn3/verl/utils/profiler/nvtx_profile.py", line 180, in wrapper
result = func(self, *args, **kwargs)
File "/root/epfs/code/verl-self-dev-flash_attn3/verl/workers/fsdp_workers.py", line 741, in generate_sequences
with self.rollout_sharding_manager:
File "/root/epfs/code/verl-self-dev-flash_attn3/verl/utils/profiler/performance.py", line 105, in f
return self.log(decorated_function, *args, **kwargs)
File "/root/epfs/code/verl-self-dev-flash_attn3/verl/utils/profiler/performance.py", line 118, in log
output = func(*args, **kwargs)
File "/root/epfs/code/verl-self-dev-flash_attn3/verl/workers/sharding_manager/fsdp_vllm.py", line 375, in enter
self.update_params(params, peft_config=peft_config)
File "/root/epfs/code/verl-self-dev-flash_attn3/verl/workers/sharding_manager/fsdp_vllm.py", line 536, in update_params
loaded_params = model.load_weights(
File "/usr/local/lib/python3.10/dist-packages/flash_rl/vllm_patch.py", line 489, in hacked_load_weights
loader._process_weights_after_loading(model, None, None)
File "/usr/local/lib/python3.10/dist-packages/flash_rl/vllm_patch.py", line 85, in hacked_process_weights_after_loading
original_process_weights_after_loading(model, model_config, target_device)
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/loader.py", line 180, in _process_weights_after_loading
quant_method.process_weights_after_loading(module)
File "/usr/local/lib/python3.10/dist-packages/flash_rl/vllm_patch.py", line 109, in hacked_kvcache_process_weights_after_loading
return original_kvcache_process_weights_after_loading(self, layer)
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/kv_cache.py", line 103, in process_weights_after_loading
if layer.q_scale > 0.0:
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1928, in getattr
raise AttributeError(
AttributeError: 'Attention' object has no attribute 'q_scale'. Did you mean: '_q_scale'?. Did you mean: '_return_value'?

[LiyuanLucasLiu]

Thanks for reporting!

For a start, can you try to run the script in the suggested environment as below:

docker run \
    --gpus all \
    --name verl \
    --shm-size=10g \
    --ipc=host \
    --ulimit memlock=-1 \
    --ulimit stack=67108864 \
    -e NCCL_P2P_LEVEL=NVL \
    -it hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.3-flashinfer0.2.2-cxx11abi0

If the issue is resolved after changing the environment, wondering whether you can share your environment information for debugging.

If the issue persists, can you try running the example script in https://github.com/yaof20/verl/blob/flash-rl/recipe/flash_rl/gsm8k_qwen0_5B_fp8.sh.

If the issue is resolved, can you share more information about your training script.

If the issue persisted, please set export FLASHRL_LOGGING_LEVEL=DEBUG and share the log : )

[LiyuanLucasLiu]

Actually, we met similar issues before. A quick fix (my guess) is implemented at: https://github.com/yaof20/Flash-RL/tree/lucas/added_q_scale (previously we patched for k_scale and v_scale, but not q_scale). can you also try this as well? Many thanks!

[kisseternity]

Actually, we met similar issues before. A quick fix (my guess) is implemented at: https://github.com/yaof20/Flash-RL/tree/lucas/added_q_scale (previously we patched for k_scale and v_scale, but not q_scale). can you also try this as well? Many thanks!

It works well on one GPU, yet another problem happens, on 8 gpus it hangs, don't know why.

[kisseternity]

Actually, we met similar issues before. A quick fix (my guess) is implemented at: https://github.com/yaof20/Flash-RL/tree/lucas/added_q_scale (previously we patched for k_scale and v_scale, but not q_scale). can you also try this as well? Many thanks!

It works well on one GPU, yet another problem happens, on 8 gpus it hangs, don't know why.

Well, this may relate to the initialization in ray.init and the patch check code, will further check it.

[kisseternity]

Actually, we met similar issues before. A quick fix (my guess) is implemented at: https://github.com/yaof20/Flash-RL/tree/lucas/added_q_scale (previously we patched for k_scale and v_scale, but not q_scale). can you also try this as well? Many thanks!

It works well on one GPU, yet another problem happens, on 8 gpus it hangs, don't know why.

Well, this may relate to the initialization in ray.init and the patch check code, will further check it.

It seems that the code hangs here

[LiyuanLucasLiu]

thanks for reporting again!

1. wondering whether you can try manually create a ray cluster, and then submit jobs? Specifically,

to create a ray cluster:

ray start --head --num-gpus 8

to submit job to ray (supposing to run at the root of the verl project):

ray job submit --runtime-env="verl/trainer/runtime_env.yaml" \
    --working-dir "${PWD}" \
    --submission-id ${ID} \
    -- python3 -m verl.trainer.main_ppo ... 

2. in our provided example as in here. The gsm8k example is actually doing a single-node multi-gpu setup. Wondering whether you can try swap the environment / the script to help pinpoint the source of the hanging.

[kisseternity]

thanks for reporting again!

1. wondering whether you can try manually create a ray cluster, and then submit jobs? Specifically,

to create a ray cluster:

ray start --head --num-gpus 8

to submit job to ray (supposing to run at the root of the verl project):

ray job submit --runtime-env="verl/trainer/runtime_env.yaml" \
    --working-dir "${PWD}" \
    --submission-id ${ID} \
    -- python3 -m verl.trainer.main_ppo ... 

2. in our provided example as in here. The gsm8k example is actually doing a single-node multi-gpu setup. Wondering whether you can try swap the environment / the script to help pinpoint the source of the hanging.

Thanks, it turns out the env variables should be passed through the yaml file instead of export. Now I can use 8 nodes to train 32B model. In hacked_process_weights_after_loading func, OOM happened while the original train code didn't.

[kisseternity]

Besides, I can't see throughput improvement during my training, wonder if the throughput improvements need some specific training setting.

[LiyuanLucasLiu]

thanks for confirming the previous issue is gone.

1. as to the throughput improvment, it only happens to large models + long response lengths. e.g., as in Figure 1 (left) in our blog, there is basically almost no speedup for 7b dense model;

2. as to the OOM issue, can you set both VERL_LOGGING_LEVEL and FLASHRL_LOGGING_LEVEL to DEBUG and report the log for debugging?

[LiyuanLucasLiu]

Some additional tips:

1. For a better throughput, we recommend using TP2 for vLLM (in our experiments, both baselines and our methods are using TP2 & are faster than TP4 setting as in the original DAPO script.

2. To reduce OOM, we find it very helpful to use actor_rollout_ref.model.use_fused_kernels=True

[LiyuanLucasLiu]

We reproduced the OOM bug w. 32B + fp8. Working on a fix now. In the meantime, you can try the int8 config for the 32B model.