TensorBoard is started as a subprocess eagerly in the trainer's __init__ (GenericTrainer.py:74-79, CloudTrainer.py:30-33, via BaseTrainer._start_tensorboard()), but it's only ever stopped in end() (GenericTrainer.py:883-884, CloudTrainer.py:116-117).
scripts/train.py calls trainer.start() and trainer.train() with no exception handling besides KeyboardInterrupt. If either raises any other exception, main() exits without reaching trainer.end(), leaving the TensorBoard subprocess (and its child server process) running as an orphan. This applies equally to local (GenericTrainer) and cloud (CloudTrainer) training.
In cloud training this is particularly disruptive: the remote training command is invoked through a shell pipeline whose completion is detected by the pipe closing. If trainer.start()/trainer.train() crashes on the remote machine, the orphaned TensorBoard process keeps that pipe open, so the remote trainer process never exits from the orchestrator's point of view — even though training itself already crashed. The job appears stuck indefinitely instead of failing.
Suggested fix: wrap trainer.start()/trainer.train() in scripts/train.py so any exception (besides the already-handled KeyboardInterrupt) stops TensorBoard before re-raising, and make BaseTrainer._stop_tensorboard() safe to call even if the subprocess was never started (e.g. via hasattr guard).
Drafted by Claude
TensorBoard is started as a subprocess eagerly in the trainer's
__init__(GenericTrainer.py:74-79,CloudTrainer.py:30-33, viaBaseTrainer._start_tensorboard()), but it's only ever stopped inend()(GenericTrainer.py:883-884,CloudTrainer.py:116-117).scripts/train.pycallstrainer.start()andtrainer.train()with no exception handling besidesKeyboardInterrupt. If either raises any other exception,main()exits without reachingtrainer.end(), leaving the TensorBoard subprocess (and its childserverprocess) running as an orphan. This applies equally to local (GenericTrainer) and cloud (CloudTrainer) training.In cloud training this is particularly disruptive: the remote training command is invoked through a shell pipeline whose completion is detected by the pipe closing. If
trainer.start()/trainer.train()crashes on the remote machine, the orphaned TensorBoard process keeps that pipe open, so the remote trainer process never exits from the orchestrator's point of view — even though training itself already crashed. The job appears stuck indefinitely instead of failing.Suggested fix: wrap
trainer.start()/trainer.train()inscripts/train.pyso any exception (besides the already-handledKeyboardInterrupt) stops TensorBoard before re-raising, and makeBaseTrainer._stop_tensorboard()safe to call even if the subprocess was never started (e.g. viahasattrguard).Drafted by Claude