Skip to content

[Bugfix][Attention] Keep model-dtype query for FlashInfer builders serving non-causal attention#49293

Open
guan404ming wants to merge 1 commit into
vllm-project:mainfrom
guan404ming:fix-noncausal-fp8-q
Open

[Bugfix][Attention] Keep model-dtype query for FlashInfer builders serving non-causal attention#49293
guan404ming wants to merge 1 commit into
vllm-project:mainfrom
guan404ming:fix-noncausal-fp8-q

Conversation

@guan404ming

@guan404ming guan404ming commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fix serving an NVFP4 target with a DFlash drafter on SM100, which fails at startup with fp8 tensor core is not supported in fa2 backend because the ModelOpt checkpoint forces FP8 KV cache. Related: #48234, #46105.

  • Keep queries in model dtype when a FlashInfer builder serves non-causal layers: trtllm-gen is causal-only, so those layers route to fa2, which cannot consume FP8 queries
  • Causal FP8 paths (target model, MTP) unchanged; relax once FlashInfer's non-causal spec-decode kernels land ([Tracking][Spec Decode] DFlash Bring-Up Tracker #46105)

Test Plan

  • Serve nvidia/Qwen3.6-27B-NVFP4 + z-lab/Qwen3.6-27B-DFlash on B200 with VLLM_USE_V2_MODEL_RUNNER=1, 8 concurrent chat streams for 5 minutes
  • Serve the same target without spec decode as a causal regression check

Test Result

  • Before: startup assert regardless of --kv-cache-dtype
  • After: 116/116 requests OK, 0 errors, spec decoding active, ~382 tok/s
  • Causal-only serving: unchanged

Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working nvidia v1

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant