Error when applying to Qwen3-30B-A3B

[bknyaz]

Using scripts/qwen/run.sh, but with

--model_name="Qwen/Qwen3-30B-A3B"
--num_average_groups=96 

results in an error after around 30 out of 48 layers processed.
For Qwen1.5 the code worked fine for me.

Seems like all experts collapsed to a single cluster?

Here is the output:

assignments: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')
cluster 0 tensor([[ 0.0459,  0.0057,  0.0073,  ...,  0.0449, -0.0208,  0.0051],
        [-0.0012, -0.0141, -0.0034,  ...,  0.0101, -0.0201,  0.0173],
        [ 0.0045, -0.0215, -0.0106,  ..., -0.0471, -0.0067, -0.0110],
        ...,
        [ 0.0625, -0.0366, -0.0204,  ..., -0.0422,  0.0013,  0.0388],
        [-0.0320, -0.0049,  0.0262,  ...,  0.0128, -0.0079, -0.0232],
        [ 0.0009,  0.0209,  0.0012,  ..., -0.0192,  0.0032, -0.0334]],
       device='cuda:0')
cluster 1 tensor([], device='cuda:0', size=(0, 2048))
cluster 2 tensor([], device='cuda:0', size=(0, 2048))
...
cluster 94 tensor([], device='cuda:0', size=(0, 2048))
cluster 95 tensor([], device='cuda:0', size=(0, 2048))
new_centers: 194560, tensor([ 0.0076, -0.0030, -0.0020,  ..., -0.0065, -0.0041, -0.0101],
device='cuda:0')
Converged! 
[HC-SMoE] Computing similarities by expert outputs...:  69%|██████▉   | 33/48 [01:09<00:31,  2.11s/it] 
Traceback (most recent call last): 
File "HC-SMoE/hcsmoe/merging-qwen.py", line 292, in <module>
Fire(run_hcsmoe)                                                            
File "lib/python3.10/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
  File "lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "HC-SMoE/hcsmoe/merging-qwen.py", line 245, in run_hcsmoe
    dom_experts = grouper.cluster_experts(model=model, dataloader=dataloader_for_merging, num_groups=num_average_groups)
  File "HC-SMoE/hcsmoe/merging/grouping_qwen.py", line 345, in cluster_experts
    dom_experts = self.group_experts_by_clustering_output(
  File "HC-SMoE/hcsmoe/merging/grouping_qwen.py", line 527, in group_experts_by_clustering_output
    dom_experts[ffn_name], label = group_experts_by_clustering(
  File "lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "HC-SMoE/hcsmoe/merging/clustering.py", line 138, in group_experts_by_clustering
    closest_expert_idx = torch.argmin(final_distances, dim=0)
IndexError: argmin(): Expected reduction dim 0 to have non-zero size.
E0919 09:57:12.538000 2227002 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 0 (pid: 2228287) of binary: bin/python3.10
Traceback (most recent call last):
  File "bin/accelerate", line 7, in <module>
    sys.exit(main())
  File "lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "lib/python3.10/site-packages/accelerate/commands/launch.py", line 1091, in launch_command
    deepspeed_launcher(args)
  File "lib/python3.10/site-packages/accelerate/commands/launch.py", line 787, in deepspeed_launcher
    distrib_run.run(args)
  File "lib/python3.10/site-packages/torch/distributed/run.py", line 883, in run
    elastic_launch(
  File "lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
hcsmoe/merging-qwen.py FAILED

There's no error when "hierarchical" clustering is used, see #3.