[WIP] ABLASTR: reuse the IGF Green's function

[ax3l]

computePhiIGF rebuilt and re-transformed its Green's function on every call. That dominates the solver: measured per call on a 128^3 grid, building costs 390 ms against an 84 ms solve on CPU, roughly three quarters of the runtime, and it does not shrink with particle count. The Green's function depends on the grid only through the cell size, so this remembers which grid the one in the solver was built for and skips the rebuild while that has not changed, keeping further ones in a small least-recently-used store for callers whose grid comes back to a size it has had before.

Green's functions move in and out of the store with std::swap, so nothing is copied. The integrated Green's function is homogeneous of degree two, verified against SumOfIntegratedPotential3D to 2.8e-13, so an entry also serves any scale reached by an exact power of two, applied to phi afterwards. Two-d mode copies rather than swaps, since there m_G_fft aliases the Green's function R2C's spectral data. Controls are ablastr.igf_cache_max_entries, igf_cache_max_bytes, igf_cache_tolerance and igf_cache_verbose, with igf_rebuild_always restoring the old behavior, all documented in the parameter list.

Who benefits depends on how steady the cell size is, and the relativistic electrostatic solver is not steady by default. It derives the longitudinal stretch from a velocity, and reconstructing it costs about $\epsilon\gamma^2$, which is a few ulp at $\gamma$ of a thousand but 1e-5 at 125 GeV; the velocity is also re-measured from the particles every step and carries sampling noise of its own. igf_cache_tolerance exists for exactly that case and defaults to $64\epsilon$, so nothing changes silently. Depends on AMReX-Codes/amrex#5627 for the accessor, and is exercised from ImpactX in BLAST-ImpactX/impactx#1621.

Measurements

beam_beam_collision on the IGF solver, which needs warpx.poisson_solver=fft and open field boundaries, 5 steps, 4 threads. At the default tolerance it reuses nothing, because of the two effects above. Raising it trades a bounded error for reuse:

igf_cache_tolerance	reused	built	Green's function share	total	max rel diff vs igf_rebuild_always
default, $64\epsilon$	0	14	30.9%	1.68 s	identical
1e-6	1	11	24.6%	1.50 s
1e-5	3	9	22.2%	1.42 s
1e-4	8	4	6.9%	1.16 s	5.3e-6
1e-3	10	2	3.6%	1.09 s	2.3e-3

The comparison is on ColliderRelevant_beam1_beam2, single threaded so it means something; ParticleNumber is identical at both settings. Build counts vary by about two between runs, since QED secondary production is thread-order dependent.

Tested through ImpactX, where the grid is chosen to repeat and the default tolerance suffices. Runs are compared against igf_rebuild_always=1 and required to agree exactly on all beam moments: with the store disabled and enabled, with a mesh fitted to the beam and with a quantized one, and on an accelerating lattice where the reference energy triples. On a constant-focusing channel with a static grid that took Green's function builds from 50 to 1 and runtime from 1.53 s to 0.37 s, and on an accelerating beam from 100 to 10 and 0.95 s to 0.41 s, byte-identically.

Two things for review. The key must never miss a change in the Green's function, since a stale one is a wrong answer rather than a slow one, so it is derived inside computePhiIGF from the cell size and the solver identity rather than accepted from callers. And twod_mode and the FFT process count are now part of the identity that decides whether the solver is rebuilt, which they were not before.

🤖 Generated with Claude Code