No acceleration in GRPO training with Eagle1 speculative decoding

[Jikai0Wang]

Hi, thanks for your great work on FastRL!
I found that Eagle1 speculative decoding successfully accelerates inference benchmarking, but does not bring any speedup in end-to-end GRPO training.

Running bash examples/bench_sd.sh:

Running bash examples/grpo_7B.sh:
Autoregressive:

+Eagle1 without draft training

I was using 8 × H800 80GB, Qwen2.5-7B-instruct+Qwen2.5-7B-Eagle-RL

[Qinghao-Hu]

Thank you for your interest in our work! We use the Base model rather than the Instruct model, and these two model types differ in their response-length behavior.

In addition, based on your log, the response_length/max is 1625 for autoregressive generation but 4270 for speculative generation, which makes the comparison uneven. In our experiments with the Base model, responses typically reach the maximum length (32K) for both modes.

Please feel free to reach out if you have any further questions.

[Jikai0Wang]

Thanks for the response! I tried switching to the base model and ran additional experiments. Below are my observations using the default settings, as well as a modified setup where I enable speculative decoding with a batch-size threshold of 16 (denoted as eagle_16).

1. Sequence length behavior difference

I noticed a clear difference in the generation length distribution:

With Eagle (batch-size threshold = 32), after around 200 training steps, the maximum output length within a batch is generally below 5000, which is noticeably different from the baseline.

In contrast, with Eagle_16, the behavior is almost the opposite: after ~200 steps, outputs frequently reach the maximum allowed length, even more aggressively than the baseline.

What mechanism causes speculative decoding (and particularly the batch-size threshold) to significantly affect the max generation length distribution?

2. Impact on PG loss

I also observed that enabling speculative decoding changes the pg_loss noticeably, although the reward statistics remain largely consistent with the baseline.

Is this expected behavior?

In practice, could this affect training stability or convergence, even if rewards are similar?

Any insights would be greatly appreciated. I’m happy to provide logs or additional statistics if needed. Thanks again for your help!

[kui253]

Thanks for the response! I tried switching to the base model and ran additional experiments. Below are my observations using the default settings, as well as a modified setup where I enable speculative decoding with a batch-size threshold of 16 (denoted as eagle_16).

1. Sequence length behavior difference

I noticed a clear difference in the generation length distribution:

With Eagle (batch-size threshold = 32), after around 200 training steps, the maximum output length within a batch is generally below 5000, which is noticeably different from the baseline.

In contrast, with Eagle_16, the behavior is almost the opposite: after ~200 steps, outputs frequently reach the maximum allowed length, even more aggressively than the baseline.

What mechanism causes speculative decoding (and particularly the batch-size threshold) to significantly affect the max generation length distribution?

2. Impact on PG loss

I also observed that enabling speculative decoding changes the pg_loss noticeably, although the reward statistics remain largely consistent with the baseline.

Is this expected behavior?

In practice, could this affect training stability or convergence, even if rewards are similar?

Any insights would be greatly appreciated. I’m happy to provide logs or additional statistics if needed. Thanks again for your help!

Did you see any speedup in the experiments you ran? We tried migrating the author's Qwen2.5-7B eagle model to vLLM and achieved an acceptance rate of 80–90%. However, after integrating it into verl, there was no speedup. In fact, the Step 1 rollout time increased from 90s to approximately 130s.

I added the spec config in the LLM init stage, expecting it would activate spec decoding mode and get speedup.

if config.spec_model_path is not None:
            print("use speculative decoding with eagle model:", config.spec_model_path)
            speculative_config={
                "method": "eagle",
                "model": config.spec_model_path,
                "draft_tensor_parallel_size": config.draft_tensor_parallel_size,
                "num_speculative_tokens": config.num_speculative_tokens,
                "disable_log_stats": False
            }
        else:
            speculative_config = None   
        self.inference_engine = LLM(
            model=model_path,
            enable_sleep_mode=config.free_cache_engine,
            tensor_parallel_size=tensor_parallel_size,
            distributed_executor_backend="external_launcher",
            dtype=config.dtype,
            enforce_eager=config.enforce_eager,
            gpu_memory_utilization=config.gpu_memory_utilization,
            disable_custom_all_reduce=True,
            skip_tokenizer_init=False,
            max_model_len=max_model_len,
            max_num_seqs=config.max_num_seqs,
            load_format=load_format,
            disable_log_stats=False,
            max_num_batched_tokens=max_num_batched_tokens,
            enable_chunked_prefill=config.enable_chunked_prefill,
            enable_prefix_caching=config.enable_prefix_caching,
            trust_remote_code=trust_remote_code,
            speculative_config=speculative_config,
            seed=config.get("seed", 0),
            **compilation_config,
            **self.lora_kwargs,
            **engine_kwargs,
        )
        kwargs = dict(
            n=1,
            logprobs=0,  # can be set to 0 and let actor to recompute
            max_tokens=config.response_length,
            repetition_penalty=config.get("repetition_penalty", 1.0),
        )

w/o spec decoding

step:1 - actor/entropy:1.4346551494672894 - actor/pg_clipfrac:0.00013830423972649442 - actor/ppo_kl:4.715117813702818e-05 - actor/pg_clipfrac_lower:0.0 - actor/kl_loss:4.8740378630895975e-05 - actor/kl_coef:0.0001 - actor/pg_loss:0.009357294637993618 - actor/grad_norm:0.7989927232265472 - perf/mfu/actor:0.0 - perf/max_memory_allocated_gb:107.92096614837646 - perf/max_memory_reserved_gb:122.25390625 - perf/cpu_memory_used_gb:60.34345626831055 - actor/lr:5e-07 - training/global_step:1 - training/epoch:0 - critic/score/mean:0.22265625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.22265625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.05008507892489433 - critic/advantages/max:2.4748666286468506 - critic/advantages/min:-1.6201815605163574 - critic/returns/mean:-0.05008507892489433 - critic/returns/max:2.4748666286468506 - critic/returns/min:-1.6201815605163574 - response_length/mean:813.6796875 - response_length/max:8192.0 - response_length/min:2.0 - response_length/clip_ratio:0.013671875 - response_length_non_aborted/mean:813.6796875 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:2.0 - response_length_non_aborted/clip_ratio:0.013671875 - response/aborted_ratio:0.0 - prompt_length/mean:128.96875 - prompt_length/max:266.0 - prompt_length/min:75.0 - prompt_length/clip_ratio:0.0 - timing_s/start_profile:0.00037842900019313674 - timing_s/generate_sequences:96.22123718261719 - timing_s/generation_timing/max:111.01653289794922 - timing_s/generation_timing/min:73.25263977050781 - timing_s/generation_timing/topk_ratio:0.25 - timing_s/gen:115.53514216199983 - timing_s/reward:5.2049488580005345 - timing_s/old_log_prob:11.678000116000476 - timing_s/ref:26.216717052000604 - timing_s/adv:0.06994709800073906 - timing_s/update_actor:41.24291002999962 - timing_s/dump_rollout_generations:0.470687431999977 - timing_s/step:200.57605594000051 - timing_s/stop_profile:4.4610000259126537e-05 - timing_per_token_ms/adv:0.00014492722880336126 - timing_per_token_ms/ref:0.05431985399348702 - timing_per_token_ms/gen:0.277326051026874 - timing_per_token_ms/update_actor:0.0854534473806339 - perf/total_num_tokens:482636 - perf/time_per_step:200.57605594000051 - perf/throughput:601.5623322262027

w spec decoding

step:1 - actor/entropy:1.6270537776872516 - actor/pg_clipfrac:0.00013595063592219958 - actor/ppo_kl:-3.5272674665520753e-05 - actor/pg_clipfrac_lower:0.0 - actor/kl_loss:5.20605327380963e-05 - actor/kl_coef:0.0001 - actor/pg_loss:0.010339929585825303 - actor/grad_norm:0.8342815339565277 - perf/mfu/actor:0.0 - perf/max_memory_allocated_gb:113.13156795501709 - perf/max_memory_reserved_gb:124.42578125 - perf/cpu_memory_used_gb:84.34381866455078 - actor/lr:5e-07 - training/global_step:1 - training/epoch:0 - critic/score/mean:0.240234375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.240234375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.050847128033638 - critic/advantages/max:2.4748666286468506 - critic/advantages/min:-1.6201815605163574 - critic/returns/mean:-0.050847128033638 - critic/returns/max:2.4748666286468506 - critic/returns/min:-1.6201815605163574 - response_length/mean:796.302734375 - response_length/max:8192.0 - response_length/min:2.0 - response_length/clip_ratio:0.013671875 - response_length_non_aborted/mean:796.302734375 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:2.0 - response_length_non_aborted/clip_ratio:0.013671875 - response/aborted_ratio:0.0 - prompt_length/mean:128.96875 - prompt_length/max:266.0 - prompt_length/min:75.0 - prompt_length/clip_ratio:0.0 - timing_s/start_profile:0.0004920009523630142 - timing_s/generate_sequences:126.51872253417969 - timing_s/generation_timing/max:153.20089721679688 - timing_s/generation_timing/min:95.02102661132812 - timing_s/generation_timing/topk_ratio:0.25 - timing_s/gen:158.47398504707962 - timing_s/reward:6.341539990156889 - timing_s/old_log_prob:14.494606057647616 - timing_s/ref:29.78212613519281 - timing_s/adv:0.08928470592945814 - timing_s/update_drafter:1.842668849043548 - timing_s/update_actor:38.662886903155595 - timing_s/dump_rollout_generations:0.46969935903325677 - timing_s/step:250.33090711431578 - timing_s/stop_profile:6.434228271245956e-05 - timing_per_token_ms/gen:0.388695766928406 - timing_per_token_ms/adv:0.0001884681352589889 - timing_per_token_ms/ref:0.06286610588360429 - timing_per_token_ms/update_actor:0.08161221031655742 - perf/total_num_tokens:473739 - perf/time_per_step:250.33090711431578 - perf/throughput:473.1127744682192

Plaftform

4*pro6000

[Jikai0Wang]

I also didn't see any acceleration. I believe two important reasons are the inconsistent rollout length before and after using speculative decoding and the excessively long reward time.

[Ximingwang-09]