something failed after Data preparation and before generate video But this will not affect subsequent inference. I would like to ask whether it will impact inference speed, what the cause of the error is, and whether it is related to the graphics card model
示例38S音频,运行多卡推理脚本,在7张A800显卡 first chunk 39S,后续 3.88S/chunk,执行完成共花费3分20S,排除第一个chunk的低速,大约也需要2分40S,和项目所说接近实时的推理(对于38S音频,理想是否应在40S内完成)有所差距。想问下可能是什么原因导致的,是否和报错/显卡型号相关?
error info like this:
2026-04-06 11:19:23.118 | INFO | main:generate:118 - Data preparation done. Start to generate video...
use_return_dict is deprecated! Use return_dict instead!
use_return_dict is deprecated! Use return_dict instead!
use_return_dict is deprecated! Use return_dict instead!
use_return_dict is deprecated! Use return_dict instead!
use_return_dict is deprecated! Use return_dict instead!
use_return_dict is deprecated! Use return_dict instead!
use_return_dict is deprecated! Use return_dict instead!
[rank1]:W0406 11:19:23.802000 437 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank1]:E0406 11:19:23.802000 437 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank6]:W0406 11:19:23.805000 442 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank6]:E0406 11:19:23.805000 442 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank3]:W0406 11:19:23.806000 439 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank3]:E0406 11:19:23.806000 439 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank4]:W0406 11:19:23.807000 440 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank4]:E0406 11:19:23.808000 440 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank0]:W0406 11:19:23.809000 436 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank0]:E0406 11:19:23.809000 436 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank5]:W0406 11:19:23.810000 441 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank5]:E0406 11:19:23.810000 441 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank2]:W0406 11:19:23.811000 438 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank2]:E0406 11:19:23.811000 438 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr(*(u0
< 0, None, False, True), **{})
something failed after Data preparation and before generate video But this will not affect subsequent inference. I would like to ask whether it will impact inference speed, what the cause of the error is, and whether it is related to the graphics card model
示例38S音频,运行多卡推理脚本,在7张A800显卡 first chunk 39S,后续 3.88S/chunk,执行完成共花费3分20S,排除第一个chunk的低速,大约也需要2分40S,和项目所说接近实时的推理(对于38S音频,理想是否应在40S内完成)有所差距。想问下可能是什么原因导致的,是否和报错/显卡型号相关?
error info like this:
2026-04-06 11:19:23.118 | INFO | main:generate:118 - Data preparation done. Start to generate video...
use_return_dictis deprecated! Usereturn_dictinstead!use_return_dictis deprecated! Usereturn_dictinstead!use_return_dictis deprecated! Usereturn_dictinstead!use_return_dictis deprecated! Usereturn_dictinstead!use_return_dictis deprecated! Usereturn_dictinstead!use_return_dictis deprecated! Usereturn_dictinstead!use_return_dictis deprecated! Usereturn_dictinstead![rank1]:W0406 11:19:23.802000 437 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank1]:E0406 11:19:23.802000 437 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank6]:W0406 11:19:23.805000 442 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank6]:E0406 11:19:23.805000 442 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank3]:W0406 11:19:23.806000 439 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank3]:E0406 11:19:23.806000 439 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank4]:W0406 11:19:23.807000 440 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank4]:E0406 11:19:23.808000 440 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank0]:W0406 11:19:23.809000 436 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank0]:E0406 11:19:23.809000 436 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank5]:W0406 11:19:23.810000 441 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank5]:E0406 11:19:23.810000 441 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr((u0
< 0, None, False, True), **{})
[rank2]:W0406 11:19:23.811000 438 site-packages/torch/fx/experimental/symbolic_shapes.py:6679] [3/0] failed during evaluate_expr(u0 <
0, hint=None, size_oblivious=True, forcing_spec=False
[rank2]:E0406 11:19:23.811000 438 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr(*(u0
< 0, None, False, True), **{})