Dynamic MOE Results and Analysis - #1

Open

reachtarunhere wants to merge 25 commits into

first-versionfrom

Commits on Nov 12, 2025

feat: Implement dynamic routing for MoE blocks

reachtarunhere
and
aider-chat-bot
committed
feat: Add dynamic routing pruning mode and config

reachtarunhere
and
aider-chat-bot
committed
feat: Add script for dynamic routing evaluation on original model

reachtarunhere
and
aider-chat-bot
committed
chore: Add debug logs for MoE layer patching and threshold

reachtarunhere
and
aider-chat-bot
committed
fix: Access dynamic routing threshold directly

reachtarunhere
and
aider-chat-bot
committed
feat: Add debug logging for dynamic MoE routing decisions

reachtarunhere
and
aider-chat-bot
committed
fix: Align dynamic routing weight normalization for stability

reachtarunhere
and
aider-chat-bot
committed
fix: Add epsilon to routing weight normalization for stability

reachtarunhere
and
aider-chat-bot
committed
fix: Resolve slicing bug that pruned all experts when top_k is 0

reachtarunhere
and
aider-chat-bot
committed
feat: Save dynamic routing expert activation report to output file

reachtarunhere
and
aider-chat-bot
committed
fix: Fix AttributeError when accessing Qwen2MoeForCausalLM layers

reachtarunhere
and
aider-chat-bot
committed
chore: Update dynamic eval script for MMLU with threshold
reachtarunhere
committed
feat: Add script to loop dynamic routing thresholds for evaluation

reachtarunhere
and
aider-chat-bot
committed

Commits on Nov 13, 2025