Skip to content

Initialize/Finalize: Pop Hooks Before Calling Them - #5648

Merged
WeiqunZhang merged 1 commit into
AMReX-Codes:developmentfrom
ax3l:topic-finalize-stack-pop-before-call
Aug 26, 2026
Merged

Initialize/Finalize: Pop Hooks Before Calling Them#5648
WeiqunZhang merged 1 commit into
AMReX-Codes:developmentfrom
ax3l:topic-finalize-stack-pop-before-call

Conversation

@ax3l

@ax3l ax3l commented Aug 26, 2026

Copy link
Copy Markdown
Member

Summary

amrex::Initialize and amrex::Finalize drained their hook stacks with top()() followed by pop(). A hook that registers another one with ExecOnInitialize/ExecOnFinalize therefore left the calling hook on top of the stack: it ran a second time, and the following pop() silently discarded the newly registered hook without ever running it.

Move the hook off the stack before calling it, so that a hook which registers another one runs exactly once and the newly registered hook runs as well.

Additional background

Found while making the ABLASTR warn manager state per AMReX cycle in BLAST-WarpX/warpx#7186: a warning recorded from a finalize hook resurrects the warn manager singleton and loses the teardown hook that would have released it again.

Checklist

The proposed changes:

  • fix a bug or incorrect behavior in AMReX
  • add new capabilities to AMReX
  • changes answers in the test suite to more than roundoff level
  • are likely to significantly affect the results of downstream AMReX users
  • include documentation in the code and/or rst files, if appropriate

`amrex::Initialize` and `amrex::Finalize` drained their hook stacks with
`top()()` followed by `pop()`. A hook that registers another one with
`ExecOnInitialize`/`ExecOnFinalize` therefore left the *calling* hook on
top of the stack: it ran a second time, and the following `pop()`
silently discarded the newly registered hook without ever running it.

Move the hook off the stack before calling it, so that a hook which
registers another one runs exactly once and the newly registered hook
runs as well.

Found while making the ABLASTR warn manager state per AMReX cycle in
BLAST-WarpX/warpx#7186: a warning recorded from
a finalize hook resurrects the warn manager singleton and loses the
teardown hook that would have released it again.
@ax3l
ax3l requested a review from WeiqunZhang August 26, 2026 17:19
@ax3l ax3l added the bug label Aug 26, 2026
@ax3l ax3l changed the title [WIP] Initialize/Finalize: Pop Hooks Before Calling Them Initialize/Finalize: Pop Hooks Before Calling Them Aug 26, 2026
@ax3l

ax3l commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

/run-hpsf-gitlab-ci

@github-actions

Copy link
Copy Markdown

GitLab CI has started at https://gitlab.spack.io/amrex/amrex/-/pipelines/1779961.

@amrex-gitlab-ci-reporter

Copy link
Copy Markdown

GitLab CI 1779961 finished with status: failed. See details at https://gitlab.spack.io/amrex/amrex/-/pipelines/1779961.

@ax3l

ax3l commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

StochasticHeatEquation_1d seems to generally fail @WeiqunZhang ? Unrelated timeout, I think?

@ax3l

ax3l commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

HPSF issue analyzed by Claude:

Unrelated to this PR — confirmed by bisecting the GitLab pipeline history.

What actually failed

Pipeline 1779961: all three GPU jobs (Nvidia-H100 SP, AMD-MI300A, Intel-PVC) died at exactly execution took longer than 1h0m0s. In each, the sole failing test is StochasticHeatEquation_1d, which never completes:

job test wall time when killed
Nvidia-H100-single-precision ***Failed 2383.62 sec
AMD-MI300A ***Failed 2668.30 sec
Intel-PVC Subprocess terminated***Exception: 2520.69 sec

ctest -j 2 starts test #80 early, it occupies one slot for the rest of the hour while the other slot finishes everything else, then the runner kills the job.

Why it is not this PR

  • The nightly schedule on plain development from the same day (job 24106057, no PR code) fails identically: 227/227 Test #80: StochasticHeatEquation_1d ***Failed 2361.67 sec.
  • Every HPSF pipeline has failed since the test landed. Last green: pipeline 1749499, 2026-08-11T03:13Z. 52ff27e (Add tools for performing Kahan summation, along with a stochastic heat equation test that uses them. #5587, which adds the test) merged 2026-08-11T06:14Z. First red: pipeline 1749700, 2026-08-11T06:25Z — same test, same hang. Unbroken red ever since.
  • Mechanically it cannot be this PR anyway: all in-tree hook registrations in Src/ are ExecOnFinalize (teardown), and the hang is mid-time-loop, thousands of seconds before amrex::Finalize is reached.

Root cause of the test itself

Tests/StochasticHeatEquation/inputs runs nstep = 20000000 + ntherm = 2000000, times ensemble = 244M steps on a 32-cell 1D domain. Per step, advance() with scheme = 1 does ~10 kernel launches plus FillBoundary/OverrideSync, and the loop additionally calls u.sum(icor_box, 0) (a blocking device reduction) and accumulate_stats every single step (nstat = 1).

That is pure launch-latency work with no arithmetic to hide it. On CPU it is fine — GitHub's GNU@13 1D Debug Fortran [tests] passes it in 354.69 s (~8 µs/step). On GPU each step costs an order of magnitude more, so it blows the 1 h budget. The Intel-PVC job builds with AMReX_MPI=OFF and hangs too, so this is kernel-launch overhead, not MPI.

It only bites HPSF because .gitlab/hpsf-gitlab-ci.yml does not set AMReX_TEST_TYPE, so it defaults to All; the GitHub GPU workflows (cuda.yml, clang.yml, gcc.yml) pass -DAMReX_TEST_TYPE=Small, which excludes it.

Suggested fix

The fix likely belongs in the test rather than in CI timeouts: a CI-sized inputs file (e.g. nstep/ntherm cut ~100x, nstat raised) registered as the ctest command, keeping the full-length inputs for manual science runs. Alternatively, gate the test off when AMReX_GPU_BACKEND is set.

@WeiqunZhang
WeiqunZhang merged commit d12ac9c into AMReX-Codes:development Aug 26, 2026
75 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants