Hey, I followed the tutorial to run two tests for validating the installation.
This seems to indicate that the patches are working, but then I am mainly concerned about the drop in performance from 0.354 to 0.342 for LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL.
2025-09-26 22:10:28,426 - numexpr.utils - INFO - Note: detected 96 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2025-09-26 22:10:28,426 - numexpr.utils - INFO - Note: NumExpr detected 96 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2025-09-26 22:10:28,426 - numexpr.utils - INFO - NumExpr defaulting to 16 threads.
DEBUG 09-26 22:10:28 [__init__.py:28] No plugins for group vllm.platform_plugins found.
DEBUG 09-26 22:10:28 [__init__.py:34] Checking if TPU platform is available.
DEBUG 09-26 22:10:28 [__init__.py:44] TPU platform is not available because: No module named 'libtpu'
DEBUG 09-26 22:10:28 [__init__.py:52] Checking if CUDA platform is available.
DEBUG 09-26 22:10:28 [__init__.py:72] Confirmed CUDA platform is available.
DEBUG 09-26 22:10:28 [__init__.py:100] Checking if ROCm platform is available.
DEBUG 09-26 22:10:28 [__init__.py:114] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 09-26 22:10:28 [__init__.py:122] Checking if HPU platform is available.
DEBUG 09-26 22:10:28 [__init__.py:129] HPU platform is not available because habana_frameworks is not found.
DEBUG 09-26 22:10:28 [__init__.py:140] Checking if XPU platform is available.
DEBUG 09-26 22:10:28 [__init__.py:150] XPU platform is not available because: No module named 'intel_extension_for_pytorch'
DEBUG 09-26 22:10:28 [__init__.py:158] Checking if CPU platform is available.
DEBUG 09-26 22:10:28 [__init__.py:180] Checking if Neuron platform is available.
DEBUG 09-26 22:10:28 [__init__.py:187] Neuron platform is not available because: No module named 'transformers_neuronx'
DEBUG 09-26 22:10:28 [__init__.py:52] Checking if CUDA platform is available.
DEBUG 09-26 22:10:28 [__init__.py:72] Confirmed CUDA platform is available.
INFO 09-26 22:10:28 [__init__.py:239] Automatically detected platform cuda.
2025-09-26 22:10:31,115 - flash_rl.vllm_patch - DEBUG - Successfully patched the _process_weights_after_loading function of vllm
2025-09-26 22:10:31,115 - flash_rl.vllm_patch - DEBUG - Successfully patched kv_cache process_weights_after_loading
2025-09-26 22:10:31,115 - flash_rl - DEBUG - Patching vllm process_weights_after_loading... status: True
2025-09-26 22:10:31,115 - flash_rl.vllm_patch - DEBUG - Successfully patched vllm
2025-09-26 22:10:31,115 - flash_rl - DEBUG - Patching the vllm LLM to enable flash_rl quantization... status: True
2025-09-26 22:10:31,341 - flash_rl.vllm_patch - DEBUG - Successfully patched vllm Fp8LinearMethod create_weights
2025-09-26 22:10:31,341 - flash_rl - DEBUG - Patching the vllm fp8 linear... status: True
2025-09-26 22:10:35,143 - numexpr.utils - INFO - Note: detected 96 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2025-09-26 22:10:35,143 - numexpr.utils - INFO - Note: NumExpr detected 96 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2025-09-26 22:10:35,143 - numexpr.utils - INFO - NumExpr defaulting to 16 threads.
DEBUG 09-26 22:10:35 [__init__.py:28] No plugins for group vllm.platform_plugins found.
DEBUG 09-26 22:10:35 [__init__.py:34] Checking if TPU platform is available.
DEBUG 09-26 22:10:35 [__init__.py:44] TPU platform is not available because: No module named 'libtpu'
DEBUG 09-26 22:10:35 [__init__.py:52] Checking if CUDA platform is available.
DEBUG 09-26 22:10:35 [__init__.py:72] Confirmed CUDA platform is available.
DEBUG 09-26 22:10:35 [__init__.py:100] Checking if ROCm platform is available.
DEBUG 09-26 22:10:35 [__init__.py:114] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 09-26 22:10:35 [__init__.py:122] Checking if HPU platform is available.
DEBUG 09-26 22:10:35 [__init__.py:129] HPU platform is not available because habana_frameworks is not found.
DEBUG 09-26 22:10:35 [__init__.py:140] Checking if XPU platform is available.
DEBUG 09-26 22:10:35 [__init__.py:150] XPU platform is not available because: No module named 'intel_extension_for_pytorch'
DEBUG 09-26 22:10:35 [__init__.py:158] Checking if CPU platform is available.
DEBUG 09-26 22:10:35 [__init__.py:180] Checking if Neuron platform is available.
DEBUG 09-26 22:10:35 [__init__.py:187] Neuron platform is not available because: No module named 'transformers_neuronx'
DEBUG 09-26 22:10:35 [__init__.py:52] Checking if CUDA platform is available.
DEBUG 09-26 22:10:35 [__init__.py:72] Confirmed CUDA platform is available.
INFO 09-26 22:10:35 [__init__.py:239] Automatically detected platform cuda.
2025-09-26 22:10:37,643 - flash_rl.vllm_patch - DEBUG - Successfully patched the _process_weights_after_loading function of vllm
2025-09-26 22:10:37,643 - flash_rl.vllm_patch - DEBUG - Successfully patched kv_cache process_weights_after_loading
2025-09-26 22:10:37,643 - flash_rl - DEBUG - Patching vllm process_weights_after_loading... status: True
2025-09-26 22:10:37,643 - flash_rl.vllm_patch - DEBUG - Successfully patched vllm
2025-09-26 22:10:37,643 - flash_rl - DEBUG - Patching the vllm LLM to enable flash_rl quantization... status: True
2025-09-26 22:10:37,869 - flash_rl.vllm_patch - DEBUG - Successfully patched vllm Fp8LinearMethod create_weights
2025-09-26 22:10:37,870 - flash_rl - DEBUG - Patching the vllm fp8 linear... status: True
2025-09-26 22:10:37,928 - datasets - DEBUG - PyTorch version 2.6.0 available.
2025-09-26 22:10:37,929 - datasets - DEBUG - Polars version 1.33.1 available.
2025-09-26 22:10:38,305 - torchao - DEBUG - Skipping import of cpp extensions: /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/torchao/_C.abi3.so: undefined symbol: _ZN3c104cuda9SetDeviceEab
2025-09-26 22:10:49,938 - lm_eval.__main__ - INFO - Selected Tasks: ['gsm8k']
2025-09-26 22:10:49,944 - lm_eval.evaluator - INFO - Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-09-26 22:10:49,944 - lm_eval.evaluator - INFO - Initializing vllm model, with arguments: {'pretrained': 'Qwen/Qwen2.5-0.5B-Instruct', 'tensor_parallel_size': 1, 'dtype': 'auto', 'gpu_memory_utilization': 0.8, 'data_parallel_size': 1}
2025-09-26 22:10:50,017 - flash_rl.vllm_patch - ERROR - flash_rl only supports external_launcher for now
2025-09-26 22:10:50,017 - flash_rl.vllm_patch - INFO - flash_rl config detected.
2025-09-26 22:10:50,017 - flash_rl.vllm_patch - INFO - Loading flash_rl config from: /lustrefs/users/nb/.flashrl_config.yaml
2025-09-26 22:10:50,018 - flash_rl.vllm_patch - INFO - Using config 0 of 1
2025-09-26 22:10:50,018 - flash_rl.vllm_patch - INFO - rank 0 flash_rl config: distributed_executor_backend: external_launcher
2025-09-26 22:10:50,018 - flash_rl.vllm_patch - INFO - rank 0 flash_rl config: fn: fp8
2025-09-26 22:10:50,018 - flash_rl.vllm_patch - INFO - rank 0 flash_rl config: load_format: auto
2025-09-26 22:10:50,018 - flash_rl.vllm_patch - INFO - rank 0 flash_rl config: model: LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL
2025-09-26 22:10:50,018 - flash_rl.vllm_patch - INFO - rank 0 flash_rl config: module_attribute_to_preserve: ['workspace']
2025-09-26 22:10:50,018 - flash_rl.vllm_patch - INFO - rank 0 flash_rl config: quantization: fp8
2025-09-26 22:10:50,018 - flash_rl.vllm_patch - WARNING - detected vLLM version 0.8.5.post1for vLLM > 0.9.1, `FlashRL` has not been tested in large scale
DEBUG 09-26 22:10:50 [__init__.py:28] No plugins for group vllm.general_plugins found.
INFO 09-26 22:10:57 [config.py:717] This model supports multiple tasks: {'classify', 'score', 'reward', 'embed', 'generate'}. Defaulting to 'generate'.
DEBUG 09-26 22:10:57 [arg_utils.py:1616] Setting max_num_batched_tokens to 16384 for LLM_CLASS usage context.
DEBUG 09-26 22:10:57 [arg_utils.py:1623] Setting max_num_seqs to 1024 for LLM_CLASS usage context.
INFO 09-26 22:10:57 [config.py:1729] Disabling V1 multiprocessing for external launcher.
INFO 09-26 22:10:57 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=16384.
INFO 09-26 22:10:58 [core.py:58] Initializing a V1 LLM engine (v0.8.5.post1) with config: model='LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL', speculative_config=None, tokenizer='LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=1234, served_model_name=LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
DEBUG 09-26 22:10:58 [decorators.py:109] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.llama.LlamaModel'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
WARNING 09-26 22:10:58 [utils.py:2522] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x14a048ee84d0>
DEBUG 09-26 22:10:58 [config.py:4110] enabled custom ops: Counter()
DEBUG 09-26 22:10:58 [config.py:4112] disabled custom ops: Counter()
DEBUG 09-26 22:10:59 [parallel_state.py:867] world_size=1 rank=0 local_rank=0 distributed_init_method=env:// backend=nccl
INFO 09-26 22:10:59 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 09-26 22:10:59 [cuda.py:221] Using Flash Attention backend on V1 engine.
INFO 09-26 22:10:59 [topk_topp_sampler.py:59] Using FlashInfer for top-p & top-k sampling.
DEBUG 09-26 22:10:59 [config.py:4110] enabled custom ops: Counter()
DEBUG 09-26 22:10:59 [config.py:4112] disabled custom ops: Counter()
INFO 09-26 22:10:59 [gpu_model_runner.py:1329] Starting to load model LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL...
DEBUG 09-26 22:10:59 [config.py:4110] enabled custom ops: Counter()
DEBUG 09-26 22:10:59 [config.py:4112] disabled custom ops: Counter({'rms_norm': 49, 'silu_and_mul': 24, 'rotary_embedding': 1})
INFO 09-26 22:10:59 [weight_utils.py:265] Using model weights format ['*.safetensors']
INFO 09-26 22:10:59 [weight_utils.py:315] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
DEBUG 09-26 22:10:59 [utils.py:218] Skipping module lm_head
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.10it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.10it/s]
INFO 09-26 22:11:00 [loader.py:458] Loading weights took 0.51 seconds
WARNING 09-26 22:11:00 [kv_cache.py:128] Using Q scale 1.0 and prob scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure Q/prob scaling factors are available in the fp8 checkpoint.
INFO 09-26 22:11:00 [gpu_model_runner.py:1347] Model loading took 0.5913 GiB and 1.071749 seconds
DEBUG 09-26 22:11:00 [decorators.py:203] Start compiling function <code object forward at 0x3d72b510, file "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 325>
DEBUG 09-26 22:11:06 [backends.py:379] Traced files (to be considered for compilation cache):
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/torch/_dynamo/polyfills/builtins.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/torch/nn/modules/container.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/torch/nn/modules/module.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/_custom_ops.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/attention/layer.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/distributed/communication_op.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/distributed/parallel_state.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/custom_op.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/layers/activation.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/layers/layernorm.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/layers/linear.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/fp8.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/w8a8_utils.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/layers/rotary_embedding.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py
DEBUG 09-26 22:11:06 [backends.py:379] /lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/platforms/interface.py
Exception in thread Thread-1 (_report_usage_worker):
Traceback (most recent call last):
File "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
self.run()
File "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/threading.py", line 1012, in run
self._target(*self._args, **self._kwargs)
File "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/usage/usage_lib.py", line 163, in _report_usage_worker
self._report_usage_once(model_architecture, usage_context, extra_kvs)
File "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/vllm/usage/usage_lib.py", line 191, in _report_usage_once
info = cpuinfo.get_cpu_info()
^^^^^^^^^^^^^^^^^^^^^^
File "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/site-packages/cpuinfo/cpuinfo.py", line 2762, in get_cpu_info
output = json.loads(output, object_hook = _utf_to_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/json/__init__.py", line 359, in loads
return cls(**kw).decode(s)
^^^^^^^^^^^^^^^^^^^
File "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/json/decoder.py", line 338, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lustrefs/users/nb/anaconda3/envs/sync-rl/lib/python3.12/json/decoder.py", line 356, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
INFO 09-26 22:11:08 [backends.py:420] Using cache directory: /lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0 for vLLM's torch.compile
INFO 09-26 22:11:08 [backends.py:430] Dynamo bytecode transform time: 7.57 s
DEBUG 09-26 22:11:08 [backends.py:86] Directly load the 0-th graph for shape None from inductor via handle ('fvs7ewzmwmvaxcnx7z5mmn5le6xaqqpdh3odylc6if5u2i3q2oaf', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/r7/cr7ktvxf2wn22twtsx2ovnccpjn6tazmwnno7b4kwrbptz4477wc.py')
DEBUG 09-26 22:11:08 [backends.py:86] Directly load the 1-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:08 [backends.py:86] Directly load the 2-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:08 [backends.py:86] Directly load the 3-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:09 [backends.py:86] Directly load the 4-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:09 [backends.py:86] Directly load the 5-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:09 [backends.py:86] Directly load the 6-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:09 [backends.py:86] Directly load the 7-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:09 [backends.py:86] Directly load the 8-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:09 [backends.py:86] Directly load the 9-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:09 [backends.py:86] Directly load the 10-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:09 [backends.py:86] Directly load the 11-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:10 [backends.py:86] Directly load the 12-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:10 [backends.py:86] Directly load the 13-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:10 [backends.py:86] Directly load the 14-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:10 [backends.py:86] Directly load the 15-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:10 [backends.py:86] Directly load the 16-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:10 [backends.py:86] Directly load the 17-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:10 [backends.py:86] Directly load the 18-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:10 [backends.py:86] Directly load the 19-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:11 [backends.py:86] Directly load the 20-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:11 [backends.py:86] Directly load the 21-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:11 [backends.py:86] Directly load the 22-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:11 [backends.py:86] Directly load the 23-th graph for shape None from inductor via handle ('fa2eocy6t2qzkoitvxxb3pihpavvimqx75idwjjrjjc6xznoofba', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/2g/c2gibby6qqfhhnujsi3e5nlrwozg5jmoasbaa45xp27ecxsz5qbq.py')
DEBUG 09-26 22:11:11 [backends.py:86] Directly load the 24-th graph for shape None from inductor via handle ('f7ofieuyrjjwmhslbq3psslynbpym42t2baoh32yhaclsw7e3g2n', '/lustrefs/users/nb/.cache/vllm/torch_compile_cache/5a3d541c89/rank_0_0/inductor_cache/me/cmewho5zbs3xxeqcd33u4srq3suq4mx2n5gbdplo6ucvqnreddx5.py')
INFO 09-26 22:11:11 [backends.py:118] Directly load the compiled graph(s) for shape None from the cache, took 2.965 s
INFO 09-26 22:11:12 [monitor.py:33] torch.compile takes 7.57 s in total
INFO 09-26 22:11:12 [kv_cache_utils.py:634] GPU KV cache size: 9,164,064 tokens
INFO 09-26 22:11:12 [kv_cache_utils.py:637] Maximum concurrency for 32,768 tokens per request: 279.67x
DEBUG 09-26 22:11:12 [backends.py:648] Warming up 1/1 for shape 512
DEBUG 09-26 22:11:12 [backends.py:659] Capturing a cudagraph for shape 512
DEBUG 09-26 22:11:13 [backends.py:648] Warming up 1/1 for shape 504
DEBUG 09-26 22:11:13 [backends.py:659] Capturing a cudagraph for shape 504
DEBUG 09-26 22:11:13 [backends.py:648] Warming up 1/1 for shape 496
DEBUG 09-26 22:11:13 [backends.py:659] Capturing a cudagraph for shape 496
DEBUG 09-26 22:11:13 [backends.py:648] Warming up 1/1 for shape 488
DEBUG 09-26 22:11:13 [backends.py:659] Capturing a cudagraph for shape 488
DEBUG 09-26 22:11:13 [backends.py:648] Warming up 1/1 for shape 480
DEBUG 09-26 22:11:13 [backends.py:659] Capturing a cudagraph for shape 480
DEBUG 09-26 22:11:14 [backends.py:648] Warming up 1/1 for shape 472
DEBUG 09-26 22:11:14 [backends.py:659] Capturing a cudagraph for shape 472
DEBUG 09-26 22:11:14 [backends.py:648] Warming up 1/1 for shape 464
DEBUG 09-26 22:11:14 [backends.py:659] Capturing a cudagraph for shape 464
DEBUG 09-26 22:11:14 [backends.py:648] Warming up 1/1 for shape 456
DEBUG 09-26 22:11:14 [backends.py:659] Capturing a cudagraph for shape 456
DEBUG 09-26 22:11:14 [backends.py:648] Warming up 1/1 for shape 448
DEBUG 09-26 22:11:14 [backends.py:659] Capturing a cudagraph for shape 448
DEBUG 09-26 22:11:14 [backends.py:648] Warming up 1/1 for shape 440
DEBUG 09-26 22:11:14 [backends.py:659] Capturing a cudagraph for shape 440
DEBUG 09-26 22:11:15 [backends.py:648] Warming up 1/1 for shape 432
DEBUG 09-26 22:11:15 [backends.py:659] Capturing a cudagraph for shape 432
DEBUG 09-26 22:11:15 [backends.py:648] Warming up 1/1 for shape 424
DEBUG 09-26 22:11:15 [backends.py:659] Capturing a cudagraph for shape 424
DEBUG 09-26 22:11:15 [backends.py:648] Warming up 1/1 for shape 416
DEBUG 09-26 22:11:15 [backends.py:659] Capturing a cudagraph for shape 416
DEBUG 09-26 22:11:15 [backends.py:648] Warming up 1/1 for shape 408
DEBUG 09-26 22:11:15 [backends.py:659] Capturing a cudagraph for shape 408
DEBUG 09-26 22:11:16 [backends.py:648] Warming up 1/1 for shape 400
DEBUG 09-26 22:11:16 [backends.py:659] Capturing a cudagraph for shape 400
DEBUG 09-26 22:11:16 [backends.py:648] Warming up 1/1 for shape 392
DEBUG 09-26 22:11:16 [backends.py:659] Capturing a cudagraph for shape 392
DEBUG 09-26 22:11:16 [backends.py:648] Warming up 1/1 for shape 384
DEBUG 09-26 22:11:16 [backends.py:659] Capturing a cudagraph for shape 384
DEBUG 09-26 22:11:16 [backends.py:648] Warming up 1/1 for shape 376
DEBUG 09-26 22:11:16 [backends.py:659] Capturing a cudagraph for shape 376
DEBUG 09-26 22:11:16 [backends.py:648] Warming up 1/1 for shape 368
DEBUG 09-26 22:11:16 [backends.py:659] Capturing a cudagraph for shape 368
DEBUG 09-26 22:11:17 [backends.py:648] Warming up 1/1 for shape 360
DEBUG 09-26 22:11:17 [backends.py:659] Capturing a cudagraph for shape 360
DEBUG 09-26 22:11:17 [backends.py:648] Warming up 1/1 for shape 352
DEBUG 09-26 22:11:17 [backends.py:659] Capturing a cudagraph for shape 352
DEBUG 09-26 22:11:17 [backends.py:648] Warming up 1/1 for shape 344
DEBUG 09-26 22:11:17 [backends.py:659] Capturing a cudagraph for shape 344
DEBUG 09-26 22:11:17 [backends.py:648] Warming up 1/1 for shape 336
DEBUG 09-26 22:11:17 [backends.py:659] Capturing a cudagraph for shape 336
DEBUG 09-26 22:11:18 [backends.py:648] Warming up 1/1 for shape 328
DEBUG 09-26 22:11:18 [backends.py:659] Capturing a cudagraph for shape 328
DEBUG 09-26 22:11:18 [backends.py:648] Warming up 1/1 for shape 320
DEBUG 09-26 22:11:18 [backends.py:659] Capturing a cudagraph for shape 320
DEBUG 09-26 22:11:18 [backends.py:648] Warming up 1/1 for shape 312
DEBUG 09-26 22:11:18 [backends.py:659] Capturing a cudagraph for shape 312
DEBUG 09-26 22:11:18 [backends.py:648] Warming up 1/1 for shape 304
DEBUG 09-26 22:11:18 [backends.py:659] Capturing a cudagraph for shape 304
DEBUG 09-26 22:11:19 [backends.py:648] Warming up 1/1 for shape 296
DEBUG 09-26 22:11:19 [backends.py:659] Capturing a cudagraph for shape 296
DEBUG 09-26 22:11:19 [backends.py:648] Warming up 1/1 for shape 288
DEBUG 09-26 22:11:19 [backends.py:659] Capturing a cudagraph for shape 288
DEBUG 09-26 22:11:19 [backends.py:648] Warming up 1/1 for shape 280
DEBUG 09-26 22:11:19 [backends.py:659] Capturing a cudagraph for shape 280
DEBUG 09-26 22:11:19 [backends.py:648] Warming up 1/1 for shape 272
DEBUG 09-26 22:11:19 [backends.py:659] Capturing a cudagraph for shape 272
DEBUG 09-26 22:11:19 [backends.py:648] Warming up 1/1 for shape 264
DEBUG 09-26 22:11:19 [backends.py:659] Capturing a cudagraph for shape 264
DEBUG 09-26 22:11:20 [backends.py:648] Warming up 1/1 for shape 256
DEBUG 09-26 22:11:20 [backends.py:659] Capturing a cudagraph for shape 256
DEBUG 09-26 22:11:20 [backends.py:648] Warming up 1/1 for shape 248
DEBUG 09-26 22:11:20 [backends.py:659] Capturing a cudagraph for shape 248
DEBUG 09-26 22:11:20 [backends.py:648] Warming up 1/1 for shape 240
DEBUG 09-26 22:11:20 [backends.py:659] Capturing a cudagraph for shape 240
DEBUG 09-26 22:11:20 [backends.py:648] Warming up 1/1 for shape 232
DEBUG 09-26 22:11:20 [backends.py:659] Capturing a cudagraph for shape 232
DEBUG 09-26 22:11:21 [backends.py:648] Warming up 1/1 for shape 224
DEBUG 09-26 22:11:21 [backends.py:659] Capturing a cudagraph for shape 224
DEBUG 09-26 22:11:21 [backends.py:648] Warming up 1/1 for shape 216
DEBUG 09-26 22:11:21 [backends.py:659] Capturing a cudagraph for shape 216
DEBUG 09-26 22:11:21 [backends.py:648] Warming up 1/1 for shape 208
DEBUG 09-26 22:11:21 [backends.py:659] Capturing a cudagraph for shape 208
DEBUG 09-26 22:11:21 [backends.py:648] Warming up 1/1 for shape 200
DEBUG 09-26 22:11:21 [backends.py:659] Capturing a cudagraph for shape 200
DEBUG 09-26 22:11:21 [backends.py:648] Warming up 1/1 for shape 192
DEBUG 09-26 22:11:21 [backends.py:659] Capturing a cudagraph for shape 192
DEBUG 09-26 22:11:22 [backends.py:648] Warming up 1/1 for shape 184
DEBUG 09-26 22:11:22 [backends.py:659] Capturing a cudagraph for shape 184
DEBUG 09-26 22:11:22 [backends.py:648] Warming up 1/1 for shape 176
DEBUG 09-26 22:11:22 [backends.py:659] Capturing a cudagraph for shape 176
DEBUG 09-26 22:11:22 [backends.py:648] Warming up 1/1 for shape 168
DEBUG 09-26 22:11:22 [backends.py:659] Capturing a cudagraph for shape 168
DEBUG 09-26 22:11:22 [backends.py:648] Warming up 1/1 for shape 160
DEBUG 09-26 22:11:22 [backends.py:659] Capturing a cudagraph for shape 160
DEBUG 09-26 22:11:23 [backends.py:648] Warming up 1/1 for shape 152
DEBUG 09-26 22:11:23 [backends.py:659] Capturing a cudagraph for shape 152
DEBUG 09-26 22:11:23 [backends.py:648] Warming up 1/1 for shape 144
DEBUG 09-26 22:11:23 [backends.py:659] Capturing a cudagraph for shape 144
DEBUG 09-26 22:11:23 [backends.py:648] Warming up 1/1 for shape 136
DEBUG 09-26 22:11:23 [backends.py:659] Capturing a cudagraph for shape 136
DEBUG 09-26 22:11:23 [backends.py:648] Warming up 1/1 for shape 128
DEBUG 09-26 22:11:23 [backends.py:659] Capturing a cudagraph for shape 128
DEBUG 09-26 22:11:23 [backends.py:648] Warming up 1/1 for shape 120
DEBUG 09-26 22:11:24 [backends.py:659] Capturing a cudagraph for shape 120
DEBUG 09-26 22:11:24 [backends.py:648] Warming up 1/1 for shape 112
DEBUG 09-26 22:11:24 [backends.py:659] Capturing a cudagraph for shape 112
DEBUG 09-26 22:11:24 [backends.py:648] Warming up 1/1 for shape 104
DEBUG 09-26 22:11:24 [backends.py:659] Capturing a cudagraph for shape 104
DEBUG 09-26 22:11:24 [backends.py:648] Warming up 1/1 for shape 96
DEBUG 09-26 22:11:24 [backends.py:659] Capturing a cudagraph for shape 96
DEBUG 09-26 22:11:24 [backends.py:648] Warming up 1/1 for shape 88
DEBUG 09-26 22:11:24 [backends.py:659] Capturing a cudagraph for shape 88
DEBUG 09-26 22:11:25 [backends.py:648] Warming up 1/1 for shape 80
DEBUG 09-26 22:11:25 [backends.py:659] Capturing a cudagraph for shape 80
DEBUG 09-26 22:11:25 [backends.py:648] Warming up 1/1 for shape 72
DEBUG 09-26 22:11:25 [backends.py:659] Capturing a cudagraph for shape 72
DEBUG 09-26 22:11:25 [backends.py:648] Warming up 1/1 for shape 64
DEBUG 09-26 22:11:25 [backends.py:659] Capturing a cudagraph for shape 64
DEBUG 09-26 22:11:25 [backends.py:648] Warming up 1/1 for shape 56
DEBUG 09-26 22:11:25 [backends.py:659] Capturing a cudagraph for shape 56
DEBUG 09-26 22:11:26 [backends.py:648] Warming up 1/1 for shape 48
DEBUG 09-26 22:11:26 [backends.py:659] Capturing a cudagraph for shape 48
DEBUG 09-26 22:11:26 [backends.py:648] Warming up 1/1 for shape 40
DEBUG 09-26 22:11:26 [backends.py:659] Capturing a cudagraph for shape 40
DEBUG 09-26 22:11:26 [backends.py:648] Warming up 1/1 for shape 32
DEBUG 09-26 22:11:26 [backends.py:659] Capturing a cudagraph for shape 32
DEBUG 09-26 22:11:26 [backends.py:648] Warming up 1/1 for shape 24
DEBUG 09-26 22:11:26 [backends.py:659] Capturing a cudagraph for shape 24
DEBUG 09-26 22:11:26 [backends.py:648] Warming up 1/1 for shape 16
DEBUG 09-26 22:11:26 [backends.py:659] Capturing a cudagraph for shape 16
DEBUG 09-26 22:11:27 [backends.py:648] Warming up 1/1 for shape 8
DEBUG 09-26 22:11:27 [backends.py:659] Capturing a cudagraph for shape 8
DEBUG 09-26 22:11:27 [backends.py:648] Warming up 1/1 for shape 4
DEBUG 09-26 22:11:27 [backends.py:659] Capturing a cudagraph for shape 4
DEBUG 09-26 22:11:27 [backends.py:648] Warming up 1/1 for shape 2
DEBUG 09-26 22:11:27 [backends.py:659] Capturing a cudagraph for shape 2
DEBUG 09-26 22:11:27 [backends.py:648] Warming up 1/1 for shape 1
DEBUG 09-26 22:11:27 [backends.py:659] Capturing a cudagraph for shape 1
INFO 09-26 22:11:28 [gpu_model_runner.py:1686] Graph capturing finished in 15 secs, took 0.48 GiB
INFO 09-26 22:11:28 [core.py:159] init engine (profile, create kv cache, warmup model) took 27.46 seconds
2025-09-26 22:11:31,577 - lm_eval.api.task - INFO - Building contexts for gsm8k on rank 0...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:03<00:00, 429.85it/s]
2025-09-26 22:11:34,659 - lm_eval.evaluator - INFO - Running generate_until requests
Processed prompts: 100%|█████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:07<00:00, 174.48it/s, est. speed input: 173234.59 toks/s, output: 21039.03 toks/s]
Running generate_until requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:07<00:00, 171.50it/s]
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
2025-09-26 22:12:02,863 - lm_eval.loggers.evaluation_tracker - INFO - Output path not provided, skipping saving results aggregated
vllm (pretrained=Qwen/Qwen2.5-0.5B-Instruct,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=1), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3397|± |0.0130|
| | |strict-match | 5|exact_match|↑ |0.2714|± |0.0122|
[rank0]:[W926 22:12:03.258731542 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Hey, I followed the tutorial to run two tests for validating the installation.
gives
score-A: 0.354gives
score-B: 0.342Setting
flashrl setup --fn fp8 --model LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL -o $HOME/.flashrl_config.yamlresults in score of around ~0.34 for both the tests.
However, when I switch to another model,
flashrl setup --fn fp8 --model Qwen/Qwen2.5-0.5B-Instruct -o $HOME/.flashrl_config.yamlthe score dropped to around ~0.31 for both the tests. Also, switching to
Qwen/Qwen3-0.6Bresults in ~0.39This seems to indicate that the patches are working, but then I am mainly concerned about the drop in performance from 0.354 to 0.342 for
LiyuanLucasLiu/Qwen2.5-0.5B-Instruct-VERL.Attaching the log for your reference: