I noticed this repo has recent activity from Damai Dai and Chengqi Deng — the same researchers behind DeepSeekMoE's expert routing architecture.
Quick question on MoE routing: has the team considered replacing the learned router W_r·x with profile-based routing? Each expert carries a calibrated benchmark vector. Router uses cosine similarity instead of learned weights. Zero router parameters (cosine sim is pure math). Experts become swappable post-training (update one array, no retraining).
At V3 scale this reduces routing params from ~106M to ~500K. At V4 scale (~384 experts) it's ~192M vs ~500K. The bias balancing mechanism remains compatible — just add b_i to cosine similarity scores before top-K.
We have a working implementation proving this (99.9% routing accuracy, 38× swap isolation, identical inference speed): https://github.com/washingtoneimae-dot/profile-moe
Would love a technical opinion from the people who built the original MoE architecture.
I noticed this repo has recent activity from Damai Dai and Chengqi Deng — the same researchers behind DeepSeekMoE's expert routing architecture.
Quick question on MoE routing: has the team considered replacing the learned router W_r·x with profile-based routing? Each expert carries a calibrated benchmark vector. Router uses cosine similarity instead of learned weights. Zero router parameters (cosine sim is pure math). Experts become swappable post-training (update one array, no retraining).
At V3 scale this reduces routing params from ~106M to ~500K. At V4 scale (~384 experts) it's ~192M vs ~500K. The bias balancing mechanism remains compatible — just add b_i to cosine similarity scores before top-K.
We have a working implementation proving this (99.9% routing accuracy, 38× swap isolation, identical inference speed): https://github.com/washingtoneimae-dot/profile-moe
Would love a technical opinion from the people who built the original MoE architecture.