2025-09-22 09:20:10 | INFO | Starting pipeline execution
...
2025-09-22 09:23:15 | INFO | Completed processing 6 file(s)
2025-09-22 09:23:55 | INFO | Completed processing 1 file(s)
2025-09-22 09:23:55 | INFO | No more files to process, staring idle time count
...
2025-09-22 09:33:48 | WARNING | Idle timeout reached, initiating shutdown
2025-09-22 09:33:48 | INFO | Shutting down pipeline
...
2025-09-22 09:33:51 | INFO | Pipeline shutdown complete
Got the following feedback:
Add a log message for informing completion of all staged files, like so:
At pipeline termination, log the total number of processed vs. failed files out of all files staged during that run, like so:
Periodically log warnings for long-pending workers (for a Slurm task), like so: