训练问题

[xyl-507]

感谢作者的杰出贡献。

我在训练过程中，发现在115epoch和219epoch会停止，显存仍在占用，但训练已经停止，而且中间的epoch也未能保存权重，请问作者有出现这种情况吗？

期待你的回复。

[zjsxlxcoming]

请问你解决了吗

[xyl-507]

没解决，只训100epoch

[suxiaolin-collab]

Traceback (most recent call last):
File "lib/train/run_training.py", line 114, in
main()
File "lib/train/run_training.py", line 106, in main
run_training(args.script, args.config, cudnn_benchmark=args.cudnn_benchmark,
File "lib/train/run_training.py", line 79, in run_training
expr_func(settings)
File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/train_script.py", line 97, in run
trainer.train(cfg.TRAIN.EPOCH, load_latest=True, fail_safe=True)
File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/trainers/base_trainer.py", line 85, in train
self.train_epoch()
File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/trainers/ltr_trainer.py", line 145, in train_epoch
self.cycle_dataset(loader)
File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/trainers/ltr_trainer.py", line 89, in cycle_dataset
loss, stats = self.actor(data)
File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/actors/ostrack.py", line 48, in call
loss, status = self.compute_losses(out_dict, data)
File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/actors/ostrack.py", line 151, in compute_losses
loss, status = self.compute_losses_CENTER(pred_dict, gt_dict,)
File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/actors/ostrack.py", line 165, in compute_losses_CENTER
raise ValueError("Network outputs is NAN! Stop Training")
ValueError: Network outputs is NAN! Stop Training
你好我在训练到285轮的时候会出现loss边NAN的情况，请问你们有没有出现过

[zjsxlxcoming]

在base_trainer.py中修改就能保存每一轮权重了   2682408561 ***@***.***  

…

------------------ 原始邮件 ------------------ 发件人: "Yuanliang Xue ***@***.***>; 发送时间: 2025年9月7日(星期天) 上午9:40 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [TsingWei/LiteTrack] 训练问题 (Issue #20) xyl-507 left a comment (TsingWei/LiteTrack#20) 没解决，只训100epoch — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[zjsxlxcoming]

不好意思，目前还没训练这么多，还没遇到这个问题   2682408561 ***@***.***  

…

------------------ 原始邮件 ------------------ 发件人: ***@***.***>; 发送时间: 2025年9月7日(星期天) 中午1:59 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [TsingWei/LiteTrack] 训练问题 (Issue #20) suxiaolin-collab left a comment (TsingWei/LiteTrack#20) Traceback (most recent call last): File "lib/train/run_training.py", line 114, in main() File "lib/train/run_training.py", line 106, in main run_training(args.script, args.config, cudnn_benchmark=args.cudnn_benchmark, File "lib/train/run_training.py", line 79, in run_training expr_func(settings) File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/train_script.py", line 97, in run trainer.train(cfg.TRAIN.EPOCH, load_latest=True, fail_safe=True) File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/trainers/base_trainer.py", line 85, in train self.train_epoch() File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/trainers/ltr_trainer.py", line 145, in train_epoch self.cycle_dataset(loader) File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/trainers/ltr_trainer.py", line 89, in cycle_dataset loss, stats = self.actor(data) File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/actors/ostrack.py", line 48, in call loss, status = self.compute_losses(out_dict, data) File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/actors/ostrack.py", line 151, in compute_losses loss, status = self.compute_losses_CENTER(pred_dict, gt_dict,) File "/opt/data/private/project/LiteTrack/lib/train/../../lib/train/actors/ostrack.py", line 165, in compute_losses_CENTER raise ValueError("Network outputs is NAN! Stop Training") ValueError: Network outputs is NAN! Stop Training 你好我在训练到285轮的时候会出现loss边NAN的情况，请问你们有没有出现过 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[kun-dragon]

Do you train with 'B4_cae_center_got10k_ep100.yaml'? Is the performance the same as that reported in the paper?

[DarlingYin]

Do you train with 'B4_cae_center_got10k_ep100.yaml'? Is the performance the same as that reported in the paper?

I cant !
Just 0.61 AO always ! I have try many settings!
cae_base_ep1600pth + GOT10K_train for 100 epoch
请问作者是用多个数据集跑完然后推理吗？ 我一直才0.61，复现不出来？Thanks！

[Jackma-Q7]

@DarlingYin 您好，请问cae_base_ep1600pth + GOT10K_train for 100 epoch B8您尝试过吗？我训练了很多次也达不到论文得70.4

[TsingWei]

@Jackma-Q7 差多少，啥device