T8Code finalizers are global MPI barriers

[vchuravy]

I am currently observing some hangs due to t8code finalizers.

x-ref: trixi-framework/Trixi.jl#2910 (comment)

Locally I managed to catch the following backtrace:

MPIC_Wait at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPIC_Sendrecv at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
Trixi.jl tests |    1      1  44m17.5s
MPIR_Barrier_intra_dissemination at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPIR_Barrier_intra_k_dissemination at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPIR_Barrier_allcomm_auto at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPIR_Barrier_impl at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPIR_Barrier_intra_smp at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPIR_Barrier_allcomm_auto at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPIR_Barrier_impl at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPID_Win_free at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
MPI_Win_free at /home/vchuravy/.julia/artifacts/117d9048128625bb3418b0b4cca48739c5d28740/lib/libmpi.so (unknown line)
sc_shmem_free at /home/vchuravy/.julia/artifacts/c4e4aead2400d536f3c56b0e7750d09c9bc9036d/lib/libsc.so.2.0.0 (unknown line)
t8_shmem_array_destroy at /home/vchuravy/.julia/artifacts/c4e4aead2400d536f3c56b0e7750d09c9bc9036d/lib/libt8.so (unknown line)
t8_forest_unref at /home/vchuravy/.julia/artifacts/c4e4aead2400d536f3c56b0e7750d09c9bc9036d/lib/libt8.so (unknown line)
t8_forest_unref at /home/vchuravy/.julia/packages/T8code/8B82X/src/Libt8.jl:16214 [inlined]
#5 at /home/vchuravy/.julia/packages/T8code/8B82X/src/T8code.jl:202
unknown function (ip: 0x7f6c5458f065)
_jl_invoke at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/gf.c:2897 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/gf.c:3079
ERROR: run_finalizer at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/gc.c:318
jl_gc_run_finalizers_in_list at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/gc.c:408
run_finalizers at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/gc.c:454
ijl_atexit_hook at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/init.c:299
ijl_exit at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/init.c:207
LoadError: exit at ./initdefs.jl:28 [inlined]
exec_options at ./client.jl:321
_start at ./client.jl:550
jfptr__start_83197.1 at /home/vchuravy/.julia/juliaup/julia-1.10.11+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/gf.c:2897 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/gf.c:3079
jl_apply at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/cli/loader_exe.c:58
unknown function (ip: 0x7f6d180276c0)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
unknown function (ip: (nil))
Allocations: 883234696 (Pool: 882185905; Big: 1048791); GC: 288

The other processes are also executing MPI barriers, so I assume rank=0 has a slightly imbalanced GC workload.

Now Julia's finalizers are "able to run at any time in any order" and so having an MPI barrier in one is a recipe for eventual hangs.

I don't have any good suggestions here, MPI and GC is a topic that has been often discussed over the last few years.

Option A

Don't do anything and tell people to run GC.gc() before MPI.Barrier

Option B

Change t8code memory management to manual instead of using finalizers, shifting the burden to the users

Option C

Use delayed finalization, e.g. still use finalizers, but add them to a queue instead and then have something like T8Code.release_unreachable_memory that the user must manually invoke. Doesn't fully fix the order problem

[benegee]

Some context: trixi-framework/Trixi.jl#2172
So I think finalizers were introduced to avoid option B.

[vchuravy]

I didn't quite see the:

ijl_atexit_hook at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/init.c:299
ijl_exit at /cache/build/builder-amdci4-2/julialang/julia-release-1-dot-10/src/init.c:207

this morning, so it indeed was a joint problem. One Trixi process was shutting down because it had encountered an error, and the two other ones were stuck in an unrelated barrier: trixi-framework/Trixi.jl#2990

Looking at the code it seems t8code.jl is trying to do option C, by using a queue and then deleting the objects together.

The backtrace is still a bit weird, since it directly calls the finalizer instead of going through clean_up.