perf(packaging): relocate onefile extraction off the system temp for MPI ranks (#67)

[FinbarArgus]

Fixes the multi-core packaging crash in issue #67, rescoped to what actually
works on the shipped PyInstaller 6.x after real-build testing (see thread).

Problem

A multi-core analysis in the packaged app runs
mpiexec -n N <bundle> --_cuflynx-run-analysis .... Each rank is a fresh onefile
exe that unpacks its own _MEIxxxxxx tree on startup, into the system $TMPDIR by
default. A full/small $TMPDIR (HPC login nodes, containers, locked-down /tmp)
becomes an opaque crash before any Python runs:
[PYI-...:ERROR] Could not create temporary directory!.

Fix — relocate the extraction off the system temp

runtime_tmpdir in the spec can't do it (6.x doesn't expand ~/$VAR on POSIX,
and the spec bakes the build machine's path). So at launch time, when the frozen
app re-invokes itself as the runner, runner_launch_env() sets
TMPDIR/TMP/TEMP to a roomy per-build user-cache dir
<user-cache>/CUFLynx/onefile-cache/<key> (key = hash of exe path+size+mtime).
Every rank then extracts there instead of a volatile $TMPDIR. Best-effort: if the
cache dir can't be created, the environment is left untouched (no regression). For
an external interpreter it's skipped entirely.

Scope — relocation, not de-duplication

Each rank still extracts its own copy; this moves where, not how many. The
first cut of this PR also tried to make the ranks share one extraction via
PyInstaller's _MEIPASS2 signal — I tested that against a real onefile build and
it's a no-op on PyInstaller 6.x (the shipped toolchain):

• the 6.21 bootloader has no _MEIPASS2 (strings confirms); parent→child
  reuse was rewritten as the internal _PYI_APPLICATION_HOME_DIR /
  _PYI_PARENT_PROCESS_LEVEL protocol;
• pointing independent mpiexec ranks at a live parent extraction via those vars
  (levels 0/1/2) still produced N distinct _MEIxxxxxx dirs.

So _MEIPASS2 is dropped. Real de-duplication would need a different approach
(e.g. a onedir analysis build) — left as a follow-up on #67.

Tests

• apps/api/tests/test_runtime_paths.py — path/key helpers + runner_launch_env
  env construction (36 passed). Asserts the relocation vars are set for the bundle,
  skipped for an external interpreter and from source, _MEIPASS2 is not set,
  and an uncreatable cache leaves the env untouched.
• Backend 338 unit passed.
• The frozen-only relocation still wants a CI frozen build to confirm end-to-end
  (documented in packaging/README.md); the destructive _MEIPASS2 claim was the
  risky part and it's gone.

🤖 Generated with Claude Code

[FinbarArgus]

⚠️ Tested against a real onefile build — the _MEIPASS2 reuse is a no-op on PyInstaller 6.x

I validated the mechanism with a minimal onefile probe built by the same
PyInstaller 6.21 this project ships (scripts/package.py pins pyinstaller>=6.0;
packaging/README.md says 6.21), driven under the same mpiexec (OpenMPI 4.1.2).

Finding 1 — _MEIPASS2 does not exist in the 6.x bootloader

strings on the built executable's bootloader:

_MEIPASS
_PYI_APPLICATION_HOME_DIR
_PYI_PARENT_PROCESS_LEVEL
PYINSTALLER_RESET_ENVIRONMENT
PYINSTALLER_STRICT_UNPACK_MODE
TMPDIR

There is no _MEIPASS2. PyInstaller 6.0 renamed the parent→child
extraction-reuse signal to _PYI_APPLICATION_HOME_DIR (+ _PYI_PARENT_PROCESS_LEVEL).
So the env var this PR sets is simply ignored by the bootloader it targets.

Finding 2 — behaviour: ranks still each extract ~1 GB

• Baseline (4 ranks, no reuse var): 4 distinct _MEIxxxxxx dirs. Expected.
• With _MEIPASS2=<parent _MEIPASS> (what this PR does): still 4 distinct
  extraction dirs — no reuse. The var is dead.
• With the correct 6.x var _PYI_APPLICATION_HOME_DIR (parent kept alive
  holding its extraction), and with _PYI_PARENT_PROCESS_LEVEL ∈ {0,1,2}: still
  4 distinct extraction dirs. Externally pointing independent mpiexec ranks
  at a shared extraction is not achievable via these env vars — the 6.x
  level-based reuse is an internal parent/child re-exec protocol, not a general
  "share this extraction" switch.

So the headline benefit (N ranks share one extraction; N× → 1× disk) does not
happen on the shipped toolchain, and it isn't a simple var-rename fix.

What does still work

TMPDIR/TMP/TEMP are honoured by the bootloader (confirmed: ranks extract
under the TMPDIR we set). So the relocation half is real — it moves extraction
off a full/small system /tmp onto the roomy per-build cache dir, which fixes the
"Could not create temporary directory!" crash symptom. But note it does not
dedupe: N ranks still create N distinct _MEIxxxxxx under that cache dir, so disk
usage stays N× — just in a roomier place.

Why the unit tests passed anyway

test_runtime_paths.py only asserts the contents of the env dict that
runner_launch_env returns — never that the bootloader honours those vars. The
one thing that mattered (does _MEIPASS2 cause reuse) can only be seen with a
frozen build, which is exactly why this was drafted pending CI — and it turns out
the answer is no.

Recommendation

Moving back to draft. Options:

1. Rescope to relocation-only — drop the _MEIPASS2/reuse claim, keep the
   TMPDIR move (real crash fix), and rewrite the description/comments/README to
   match. Smaller but honest win.
2. Pursue real sharing — likely needs a different approach (e.g. a onedir
   build for the analysis path so there's nothing to extract per rank, or a
   custom shared-extraction step), not env vars. Bigger.
3. Drop.

Happy to take whichever direction.