Unable to Stop the Job thread after the No Progress Issue is Observed for 120 sec and It gets stuck in a loop

[AnkitTirkey11]

20250918 051916 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051916 02m01.12s (dt j:1 t:1): No progress made for record 521 (lba 66560, offset 34078720) during read() on /t/ontap_10_97_5_166_fg_5_aws_86247937_86810038_aws_svm_0_test_qtree_179/test_file_544_529MB for 119 seconds! (01m59.00s) Since: Thu Sep 18 05:17:17 2025\r\n
440   20250918 051917 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051917 02m02.12s (dt j:1 t:1): No progress made for record 521 (lba 66560, offset 34078720) during read() on /t/ontap_10_97_5_166_fg_5_aws_86247937_86810038_aws_svm_0_test_qtree_179/test_file_544_529MB for 120 seconds! (02m00.00s) Since: Thu Sep 18 05:17:17 2025\r\n
441   20250918 051918 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051918 02m03.13s (dt j:1 t:1): No progress made for record 521 (lba 66560, offset 34078720) during read() on /t/ontap_10_97_5_166_fg_5_aws_86247937_86810038_aws_svm_0_test_qtree_179/test_file_544_529MB for 121 seconds! (02m01.00s) Since: Thu Sep 18 05:17:17 2025\r\n
442   20250918 051918 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051918 02m03.13s (dt j:1 t:1): This requests' elapsed time of 121, has exceeded the noprogtt of 120 seconds!\r\n
443   20250918 051918 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051918 02m03.13s (dt j:1 t:1):    The current time is: 1758187158 seconds => Thu Sep 18 05:19:18 2025\r\n
444   20250918 051918 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051918 02m03.13s (dt j:1 t:1): The initiated time was: 1758187037 seconds => Thu Sep 18 05:17:17 2025\r\n
445   20250918 051918 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051918 02m03.13s (dt j:1 t:1): ERROR: Error number 1 occurred on Thu Sep 18 05:19:18 2025\r\n
446   20250918 051918 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051918 02m03.13s (dt j:1 t:1): ERROR: No triggers or noprog triggers are not enabled, so stopping this job and its' threads...\r\n
447   20250918 051918 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 051918 02m03.15s (dt j:0 t:0): Job 1 is being stopped (1 thread)\r\n
448   20250918 052219 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052219 05m03.33s (dt j:0 t:0): Job 1, thread has exceeded the max terminate wait time of 180 seconds!\r\n
449   20250918 052219 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052219 05m03.34s (dt j:0 t:0): Threads have NOT terminated for 181 seconds, perhaps too slow or hung?\r\n
450   20250918 052219 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052219 05m03.34s (dt j:0 t:0): ERROR: Error number 1 occurred on Thu Sep 18 05:22:19 2025\r\n
451   20250918 052219 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052219 05m03.34s (dt j:0 t:0): ERROR: Job 1 has NOT terminated, so cancelling all threads!\r\n
452   20250918 052219 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052219 05m03.34s (dt j:0 t:0): Canceling thread number 1, thread ID: 0x7f45b1dd8700...\r\n
453   20250918 052220 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052220 05m04.34s (dt j:0 t:0): Job 1, thread has exceeded the max terminate wait time of 180 seconds!\r\n
454   20250918 052220 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052220 05m04.34s (dt j:0 t:0): Threads have NOT terminated for 182 seconds, perhaps too slow or hung?\r\n
455   20250918 052220 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052220 05m04.34s (dt j:0 t:0): ERROR: Error number 2 occurred on Thu Sep 18 05:22:20 2025\r\n
456   20250918 052220 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052220 05m04.34s (dt j:0 t:0): ERROR: Job 1 has NOT terminated, so cancelling all threads!\r\n
457   20250918 052220 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052220 05m04.34s (dt j:0 t:0): Canceling thread number 1, thread ID: 0x7f45b1dd8700...\r\n
458   20250918 052221 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052221 05m05.34s (dt j:0 t:0): Job 1, thread has exceeded the max terminate wait time of 180 seconds!\r\n
459   20250918 052221 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052221 05m05.34s (dt j:0 t:0): Threads have NOT terminated for 183 seconds, perhaps too slow or hung?\r\n
460   20250918 052221 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052221 05m05.34s (dt j:0 t:0): ERROR: Error number 3 occurred on Thu Sep 18 05:22:21 2025\r\n
461   20250918 052221 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052221 05m05.34s (dt j:0 t:0): ERROR: Job 1 has NOT terminated, so cancelling all threads!\r\n
462   20250918 052221 [MainThread:__init__:109]  [INFO] <<aws-86247937-86810038-aws-linux-2#login| 20250918 052221 05m05.34s (dt j:0 t:0): Canceling thread number 1, thread ID: 0x7f45b1dd8700...\r\n

How do we stop this job thread cancellation loop ?

[RobinTMiller]

Please include the dt version when reporting issues. The command line is often useful as well, thanks!

I'm guessing there has been a disruption which is keeping I/O from completing. Internally, dt sets a terminate state, then gives jobs the opportunity to exit cleanly up to the job wait time. But that said, when I/O is hung hard then it never completes. NFS hard mounts exhibit this behaviour along with some host I/O storage stacks.

Other than dt trying to cancel job threads, which won't help with hard I/O hangs, sending a signal like SIGQUIT will terminate the I/O monitoring thread and all job threads. But dt may still hang since the kernel I/O rundown logic may not be able to terminate hung I/O either. In these cases, even a "kill -9" won't stop dt processes.

There is no other special logic to handle hard hangs beyond thread cancel. In fact, unless I/O is resumed, oftentimes the host OS must be rebooted to clear up hung dt processes.

In your case, a trigger script could be used wen the noprog trigger time is exceeded, to try "pkill dt" if on Linux, but that would kill all dt processes, of course.