Test Issue

[Ch3ngY1]

Issue Description

The beginning of issue.

Failed Cases

HuggingFaceID,Architecture,System,Backend,Version,Mode,FailedBranch
Qwen/Qwen3-235B-A22B,Qwen3MoeForCausalLM,gb200,trtllm,1.2.0rc6,agg,automated/update-support-matrix-main
Qwen/Qwen3-235B-A22B,Qwen3MoeForCausalLM,gb200,trtllm,1.2.0rc6,disagg,automated/update-support-matrix-main

Example ErrMsg

"src/aiconfigurator/sdk/backends/trtllm_backend.py", line 90, in run_agg
    summary = self._agg_cache[isl][osl][b][ctx_tokens][max_seq_len][max_num_tokens][free_gpu_memory_fraction]
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 0.9

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/aiconfigurator/sdk/pareto_analysis.py", line 114, in agg_pareto
    summary = sess.find_best_agg_result_under_constraints(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/inference_session.py", line 128, in find_best_agg_result_under_constraints
    return self._backend.find_best_agg_result_under_constraints(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/backends/trtllm_backend.py", line 538, in find_best_agg_result_under_constraints
    summary = self.run_agg(
              ^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/backends/trtllm_backend.py", line 270, in run_agg
    mix_step_latency_ms, mix_step_energy_wms = _get_mix_step_latency(
                                               ^^^^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/backends/trtllm_backend.py", line 147, in _get_mix_step_latency
    summary = self.run_static(
              ^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/backends/base_backend.py", line 223, in run_static
    ) = self._run_static_breakdown(model, database, runtime_config, mode, stride, latency_correction_scale)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/backends/base_backend.py", line 131, in _run_static_breakdown
    context_latency_dict, context_energy_wms_dict = self._run_context_phase(
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/backends/base_backend.py", line 55, in _run_context_phase
    result = op.query(
             ^^^^^^^^^
  File "src/aiconfigurator/sdk/operations.py", line 509, in query
    result = database.query_moe(
             ^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/perf_database.py", line 5522, in query_moe
    return self._query_silicon_or_hybrid(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/perf_database.py", line 3694, in _query_silicon_or_hybrid
    return get_silicon()
           ^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/perf_database.py", line 5454, in get_silicon
    if num_tokens > token_points[-1]:
                    ~~~~~~~~~~~~^^^^
IndexError: list index out of range Failed to query moe data for num_tokens=128.0, hidden_size=2048, inter_size=768, topk=8, num_experts=128, moe_tp_size=8, moe_ep_size=1, quant_mode=<MoEQuantMode.float16: QuantMapping(memory=2, compute=1, name='float16')>, workload_distribution='power_law_1.2'. Consider using HYBRID mode.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "tools/support_matrix/support_matrix.py", line 208, in run_single_test
    result = runner.run(task_config)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/task.py", line 1416, in run
    result = self.run_agg(task_config.config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/task.py", line 1183, in run_agg
    result_df = pa.agg_pareto(
                ^^^^^^^^^^^^^^
  File "src/aiconfigurator/sdk/pareto_analysis.py", line 158, in agg_pareto
    raise RuntimeError(
RuntimeError: No results found for any parallel configuration. Showing last exception: list index out of range Failed to query moe data for num_tokens=128.0, hidden_size=2048, inter_size=768, topk=8, num_experts=128, moe_tp_size=8, moe_ep_size=1, quant_mode=<MoEQuantMode.float16: QuantMapping(memory=2, compute=1, name='float16')>, workload_distribution='power_law_1.2'. Consider using HYBRID mode.

Modified post description

[tianhaox]

We need to add it here. If it doesn't introduce any brand new operations, we can add it with one line here. For example, gpt-oss introduces attention sinks which we didn't cover it before, we need to add support for that new op and then add the model support. If the model only uses differenct num layers and different qkv heads for GQA, MQA, MHA, we think it doesn't introduce any new operations, you can leverage current implementation directly.

[tianhaox]

Which model you would like to add?

[Ch3ngY1]

GLM-4.5-Air

[Ch3ngY1]

I added the corresponding config here, like
'GLM_4.5_Air':['MOE',46,96,8,128,4096,10944,151552,131072, 8, 128, 1408, None]

But when I run the following command aiconfigurator cli --model GLM_4.5_Air --total_gpus 8 --system h200_sxm --ttft 500 --tpot 50 --isl 1000 --osl 100, I encountered this error message:

ERROR 2025-08-18 17:57:23,199 pareto_analysis.py:159] Error getting candidate workers with parallel config: tp=8, pp=1, dp=1, moe_tp=1, moe_ep=8, skip this combination: Traceback (most recent call last):
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/backends/trtllm_backend.py", line 40, in run_ifb
    summary = self._ifb_cache[isl][osl][b][ctx_tokens]
KeyError: 512

During handling of the above exception, another exception occurred:
...
ERROR 2025-08-18 17:57:23,224 inference_session.py:359] Error getting candidate workers with parallel config: tp=8, pp=1, dp=1, moe_tp=1, moe_ep=8 skip this combination: Traceback (most recent call last):
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/inference_session.py", line 353, in get_summary_df
    summary = sess.run_static(mode=mode, runtime_config=overwritten_runtime_config)
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/inference_session.py", line 57, in run_static
    return self._backend.run_static(self._model, self._database, runtime_config, mode, stride)
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/backends/base_backend.py", line 85, in run_static
    generation_latency_dict = _run_generation(batch_size, beam_width, isl, osl, stride)
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/backends/base_backend.py", line 62, in _run_generation
    latency = op.query(database, x=batch_size*beam_width, batch_size=batch_size, beam_width=beam_width, s=isl+i+1)
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/operations.py", line 137, in query
    return database.query_moe(num_tokens=x,
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/perf_database.py", line 1068, in query_moe
    num_left, num_right = self._nearest_1d_point_helper(num_tokens, list(moe_dict.keys()), inner_only=False)
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/perf_database.py", line 614, in _nearest_1d_point_helper
    assert(values is not None and len(values) >= 2), f"values is None or len(values) < 2"
AssertionError: values is None or len(values) < 2

ERROR 2025-08-18 17:57:23,224 main.py:1032] Configuration failed: 'NoneType' object has no attribute 'get_summary_df'
ERROR 2025-08-18 17:57:23,224 main.py:1033] Full traceback
Traceback (most recent call last):
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/cli/main.py", line 1024, in main
    aiconfigurator_result = aiconfigurator.run(aiconfigurator_config)
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/cli/main.py", line 145, in run
    raw_agg_df, raw_disagg_df = self._run_pareto_analysis(
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/cli/main.py", line 329, in _run_pareto_analysis
    disagg_df = pareto_analysis.disagg_pareto(
  File "/root/miniforge3/lib/python3.10/site-packages/aiconfigurator/sdk/pareto_analysis.py", line 267, in disagg_pareto
    return summary.get_summary_df()
AttributeError: 'NoneType' object has no attribute 'get_summary_df'
INFO 2025-08-18 17:57:23,225 main.py:1037] Configuration completed in 13.36 seconds

[tianhaox]

I see. MoE needs collecting relevant data for each new moe model if of different [hiddensize, intersize, experts, topk]. This is not that compatible like dense models. You need to collect moe data then.

1. prepare collector/collect.py

only keep this part, you don't have to collect other data.

modify this collect_moe_pre_0_20.py, only keeps your GLM air model's config.

3. prepare env with tensorrt_llm==0.20.0 and lock the GPU freq, enable persistent mode. then run python3 collect.py

4. add newly generated data (moe_perf.txt) to src/aiconfigurator/systems/data/h200_sxm/trtllm/0.20.0/moe_perf.txt

5. rebuild and try the command

[tianhaox]

added one a doc for adding a custom model, doc please reopen or just create a new pr for a specific model