跑T-Eval时出问题

[baowj-678]

参考teval编写代码评测本地的Qwen3-1.7B。
控制台报错如下：

outputs/teval/20250904_030834/logs/eval/Qwen3-1.7B/teval-retrieve_str_v2_zh.out
launch OpenICLEval[Qwen3-1.7B/teval-understand_str_v2_zh] on GPU 2                                                                         
 94%|███████████████████████████████████████████████████████████████████████████████████████████████▋      | 15/16 [01:50<00:07,  7.44s/it]09/04 03:10:42 - OpenCompass - ERROR - /workspace/compass/opencompass-main/opencompass/runners/local.py - _launch - 241 - task OpenICLEval[Qwen3-1.7B/teval-understand_str_v2_zh] fail, see
outputs/teval/20250904_030834/logs/eval/Qwen3-1.7B/teval-understand_str_v2_zh.out
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [01:57<00:00,  7.37s/it]
09/04 03:10:42 - OpenCompass - ERROR - /workspace/compass/opencompass-main/opencompass/runners/base.py - summarize - 64 - OpenICLEval[Qwen3-1.7B/teval-instruct_v2] failed with code 1
09/04 03:10:42 - OpenCompass - ERROR - /workspace/compass/opencompass-main/opencompass/runners/base.py - summarize - 64 - OpenICLEval[Qwen3-1.7B/teval-plan_json_v2] failed with code 1

outputs/teval/20250904_030834/logs/eval/Qwen3-1.7B/teval-instruct_v2.out报错如下：

^MMap:   0%|          | 0/553 [00:00<?, ? examples/s]^MMap: 100%|██████████| 553/553 [00:00<00:00, 17935.46 examples/s]
^MMap:   0%|          | 0/553 [00:00<?, ? examples/s]^MMap: 100%|██████████| 553/553 [00:00<00:00, 18000.75 examples/s]
Traceback (most recent call last):
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_eval.py", line 466, in <module>
    inferencer.run()
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_eval.py", line 88, in run
    self._score()
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_eval.py", line 94, in _score
    pred_dicts, pred_strs = self._load_predictions()
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_eval.py", line 140, in _load_predictions
    raise FileNotFoundError(
FileNotFoundError: Prediction files not found: neither outputs/teval/20250904_030834/predictions/Qwen3-1.7B/teval-instruct_v2.json nor outputs/teval/20250904_030834/predictions/Qwen3-1.7B/teval-instruct_v2_0.json exists

outputs/teval/20250904_030834/logs/infer/Qwen3-1.7B/teval-instruct_v2.out报错如下：

09/04 03:08:37 - OpenCompass - INFO - Task [Qwen3-1.7B/teval-instruct_v2,Qwen3-1.7B/teval-plan_json_v2,Qwen3-1.7B/teval-plan_str_v2,Qwen3-1.7B/teval-review_str_v2,Qwen3-1.7B/teval-reason_retrieve_understand_json_v2,Qwen3-1.7B/teval-reason_str_v2,Qwen3-1.7B/teval-retrieve_str_v2,Qwen3-1.7B/teval-understand_str_v2,Qwen3-1.7B/teval-instruct_v2_zh,Qwen3-1.7B/teval-plan_json_v2_zh,Qwen3-1.7B/teval-plan_str_v2_zh,Qwen3-1.7B/teval-review_str_v2_zh,Qwen3-1.7B/teval-reason_retrieve_understand_json_v2_zh,Qwen3-1.7B/teval-reason_str_v2_zh,Qwen3-1.7B/teval-retrieve_str_v2_zh,Qwen3-1.7B/teval-understand_str_v2_zh]
^MLoading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]^MLoading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.03it/s]^MLoading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  2.03it/s]^MLoading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  1.77it/s]
09/04 03:08:42 - OpenCompass - INFO - using stop words: ['<|im_end|>', '<|im_start|>', '<|endoftext|>']
^MMap:   0%|          | 0/553 [00:00<?, ? examples/s]^MMap: 100%|██████████| 553/553 [00:00<00:00, 17007.38 examples/s]
^MMap:   0%|          | 0/553 [00:00<?, ? examples/s]^MMap: 100%|██████████| 553/553 [00:00<00:00, 17467.84 examples/s]
09/04 03:08:43 - OpenCompass - INFO - Start inferencing [Qwen3-1.7B/teval-instruct_v2]
[2025-09-04 03:08:43,625] [opencompass.openicl.icl_inferencer.icl_chat_inferencer] [INFO] Starting inference process...
^M  0%|          | 0/553 [00:00<?, ?it/s]09/04 03:08:43 - OpenCompass - INFO - inputs {origin_prompt}, size: 1
^M  0%|          | 0/553 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_infer.py", line 161, in <module>
    inferencer.run()
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_infer.py", line 89, in run
    self._inference()
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_infer.py", line 134, in _inference
    inferencer.inference(retriever,
  File "/workspace/compass/opencompass-main/opencompass/openicl/icl_inferencer/icl_chat_inferencer.py", line 249, in inference
    self.infer_last(chat, index, output_handler)
  File "/workspace/compass/opencompass-main/opencompass/openicl/icl_inferencer/icl_chat_inferencer.py", line 334, in infer_last
    output = self.model.generate_from_template(
  File "/workspace/compass/opencompass-main/opencompass/models/base.py", line 201, in generate_from_template
    return self.generate(inputs, max_out_len=max_out_len, **kwargs)
  File "/workspace/compass/opencompass-main/opencompass/models/huggingface_above_v4_33.py", line 430, in generate
    messages = _convert_chat_messages(inputs)
  File "/workspace/compass/opencompass-main/opencompass/models/huggingface_above_v4_33.py", line 72, in _convert_chat_messages
    role = {
KeyError: 'user'

eval.py内容如下：

09/04 03:29:39 - OpenCompass - INFO - Task [Qwen3-1.7B/teval-instruct_v2,Qwen3-1.7B/teval-plan_json_v2,Qwen3-1.7B/teval-plan_str_v2,Qwen3-1.7B/teval-review_str_v2,Qwen3-1.7B/teval-reason_retrieve_understand_json_v2,Qwen3-1.7B/teval-reason_str_v2,Qwen3-1.7B/teval-retrieve_str_v2,Qwen3-1.7B/teval-understand_str_v2,Qwen3-1.7B/teval-instruct_v2_zh,Qwen3-1.7B/teval-plan_json_v2_zh,Qwen3-1.7B/teval-plan_str_v2_zh,Qwen3-1.7B/teval-review_str_v2_zh,Qwen3-1.7B/teval-reason_retrieve_understand_json_v2_zh,Qwen3-1.7B/teval-reason_str_v2_zh,Qwen3-1.7B/teval-retrieve_str_v2_zh,Qwen3-1.7B/teval-understand_str_v2_zh]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  1.74it/s]
09/04 03:29:44 - OpenCompass - INFO - using stop words: ['<|endoftext|>', '<|im_start|>', '<|im_end|>']
Map: 100%|██████████| 553/553 [00:00<00:00, 17395.01 examples/s]
Map: 100%|██████████| 553/553 [00:00<00:00, 17736.73 examples/s]
09/04 03:29:46 - OpenCompass - INFO - Start inferencing [Qwen3-1.7B/teval-instruct_v2]
[2025-09-04 03:29:46,093] [opencompass.openicl.icl_inferencer.icl_chat_inferencer] [INFO] Starting inference process...
  0%|          | 0/553 [00:00<?, ?it/s]09/04 03:29:46 - OpenCompass - INFO - inputs You have access to the following API:
{'name': 'FilmDouban.print_detail', 'description': 'prints the details of a movie, giving its name', 'required_parameters': [{'name': 'film_name', 'type': 'STRING', 'description': "target movie's name, must be in Chinese"}], 'optional_parameters': [], 'return_data': [{'name': 'description', 'description': 'the description of the movie'}]}
Please directly generate the response in JSON format. Do NOT add comments (//) in the response.
```
{
    // goal to call this action
    "goal": string;
    // api name to call
    "name": string;
    // input params required by current action
    "args": Record<string, any>;
}
```Call the function FilmDouban.print_detail with the parameter as follows: 'film_name' is '驯龙高手'., size: 1
  0%|          | 0/553 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_infer.py", line 161, in <module>
    inferencer.run()
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_infer.py", line 89, in run
    self._inference()
  File "/workspace/compass/opencompass-main/opencompass/tasks/openicl_infer.py", line 134, in _inference
    inferencer.inference(retriever,
  File "/workspace/compass/opencompass-main/opencompass/openicl/icl_inferencer/icl_chat_inferencer.py", line 249, in inference
    self.infer_last(chat, index, output_handler)
  File "/workspace/compass/opencompass-main/opencompass/openicl/icl_inferencer/icl_chat_inferencer.py", line 334, in infer_last
    output = self.model.generate_from_template(
  File "/workspace/compass/opencompass-main/opencompass/models/base.py", line 201, in generate_from_template
    return self.generate(inputs, max_out_len=max_out_len, **kwargs)
  File "/workspace/compass/opencompass-main/opencompass/models/huggingface_above_v4_33.py", line 430, in generate
    messages = _convert_chat_messages(inputs)
  File "/workspace/compass/opencompass-main/opencompass/models/huggingface_above_v4_33.py", line 72, in _convert_chat_messages
    role = {
KeyError: 'user'

[WangHaoZhe]

遇到了同样的问题，我是出现在跑MMLU时，跑wikitext2时却没有报错

[WangHaoZhe]

找到原因了，infer的时候就已经报错了导致eval阶段没有json可以分析。需要根据infer文件夹中的.out日志来分析报错

[baowj-678]

哦哦哦，好像是的，infer没有输出，好久没搞了