Skip to content

fix: respect explicit config.head_dim in cnets attention#342

Open
AchuthReddy-16 wants to merge 1 commit into
SafeAILab:mainfrom
AchuthReddy-16:fix-315-qwen3-head-dim
Open

fix: respect explicit config.head_dim in cnets attention#342
AchuthReddy-16 wants to merge 1 commit into
SafeAILab:mainfrom
AchuthReddy-16:fix-315-qwen3-head-dim

Conversation

@AchuthReddy-16

Copy link
Copy Markdown

Fixes #315

Credit to @gcy-hw95, who diagnosed this and tested the fix in #315 — this PR just submits it properly since it never got merged.

This PR fixes the head_dim handling in eagle/model/cnets.py.

Previously, LlamaAttention always computed:

self.head_dim = self.hidden_size // self.num_heads

That works for standard Llama-style configs, but it does not hold for some Qwen3 configs where head_dim is explicitly defined. For example, Qwen3-4B has hidden_size=2560 and num_attention_heads=32, which gives 80 using the old formula, but the actual head_dim is 128.

Because of this mismatch, loading Qwen3 checkpoints can fail with shape mismatch errors.

This PR changes the code to:

  • use config.head_dim when it exists
  • fall back to hidden_size // num_heads when config.head_dim is not present
  • remove the assertion that required hidden_size == num_heads * head_dim
  • update the attention output reshape to use num_heads * head_dim instead of assuming it is equal to hidden_size

This follows the same general pattern already used in modeling_qwen3_kv.py.

One note: I left the pretraining_tp > 1 path unchanged because it seems separate from this issue and was not part of the original fix discussed in #315.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix dimension mismatch error when reproducing Eagle3-Qwen3-4B (hidden_size != head_dim * num_heads)

1 participant