Skip to content

E3: qwen3_5_moe (linear attention + MoE FFN) #118

Description

@tlkahn

qwen3_5_moe (MODEL_QWEN3_5_MOE)

Linear-attention layers + MoE FFN. Separate enum from dense MODEL_QWEN3_5 (decision 4).

Spec: doc/plans/engine-inference-core.md § E3

Part of: #106 (E3) · Epic: #56
Depends on: E1 MoE block (#104); hybrid cache + unified gate

Tasks

  • Linear-attention layers: separate QKV/Z/A/B projections per LinearAttnWeights
  • MoE FFN via shared fwd_moe
  • Layer-type mixing / config consumption already partially present from dense path
  • Weight mapping + weights_expected_names() for MODEL_QWEN3_5_MOE
  • Support gate allow-list entry
  • Smoke generate on canonical MoE checkpoint

Notes

  • Do not collapse with MODEL_QWEN3_5 dense

Metadata

Metadata

Assignees

Labels

engine-inferenceEngine inference core (issue #51) feature areaenhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions